C compilers bottoming out in scripts may not cache correctly

[alexcrichton]

This is sort of similar to #87 except for C compilers. First discovered here we had a situation that looked like:

• Two distinct toolchains (but built by the same entity) were installed in two different locations
• Two different compilers, mips-openwrt-linux-gcc and mipsel-openwrt-linux-gcc, were used
• Both compilers were a trail of symlinks bottoming out in a shell scripts (but shell scripts at two different paths on the filesystem)
• Both shell scripts, however, were exactly the same! They used dispatch based on $0 to figure out where to go (and is why the shell script operated differently in the two locations)
• Inside of sccache the executable_digest part of the hash key was then the same for both compilers

As a consequence this meant that the mips object files (built first) were used for the mipsel compiler, causing our issues!

I'm not really sure what the best way to solve this would be :(

[luser]

Oh, this sucks. :-( gcc has a --print-prog-name= option that sounds almost useful, but it doesn't seem to do anything useful if you pass it cc or gcc:

luser@eye7:/build$ gcc --print-prog-name=gcc
gcc
luser@eye7:/build$ gcc-5 --print-prog-name=gcc-5
gcc-5
luser@eye7:/build$ gcc --print-prog-name=cc
cc
luser@eye7:/build$ gcc-5 --print-prog-name=cc
cc

clang's implementation of this seems to be more useful:

luser@eye7:/build$ clang --print-prog-name=clang
/usr/bin/clang
luser@eye7:/build$ PATH=/build/clang+llvm-5.0.1-x86_64-linux-gnu-ubuntu-16.04/bin/:$PATH clang --print-prog-name=clang
/build/clang+llvm-5.0.1-x86_64-linux-gnu-ubuntu-16.04/bin/clang

gcc's -print-search-dirs option does seem to include an install: key which might be usable:

luser@eye7:/build$ gcc -print-search-dirsinstall: /usr/lib/gcc/x86_64-linux-gnu/7/
programs: =/usr/lib/gcc/x86_64-linux-gnu/7/:/usr/lib/gcc/x86_64-linux-gnu/7/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/7/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/7/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../../x86_64-linux-gnu/bin/
libraries: =/usr/lib/gcc/x86_64-linux-gnu/7/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/7/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../../x86_64-linux-gnu/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/7/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/:/lib/x86_64-linux-gnu/7/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/7/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../../x86_64-linux-gnu/lib/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../:/lib/:/usr/lib/
luser@eye7:/build$ gcc-5 -print-search-dirs
install: /usr/lib/gcc/x86_64-linux-gnu/5/
programs: =/usr/lib/gcc/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../../x86_64-linux-gnu/bin/
libraries: =/usr/lib/gcc/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../../x86_64-linux-gnu/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/:/lib/x86_64-linux-gnu/5/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/5/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../../x86_64-linux-gnu/lib/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../:/lib/:/usr/lib/

With the toolchains in question, do you get useful output from that? If so we could fix sccache to use this as a cache input somehow, at the cost of an additional gcc invocation per compile (but presumably these -print-... commands are fast). Making it conditional on a heuristic that checks whether the compiler binary actually starts with a shebang might be good enough?

[alexcrichton]

Ah unfortunately for whatever reason the shell scripts for the musl compilers that collided in rust-lang/rust#48647 both don't work with --print-prog-name:

root@e95d6cbf1fcf:/build# mipsel-openwrt-linux-gcc --print-prog-name=gcc
gcc
root@e95d6cbf1fcf:/build# mipsel-openwrt-linux-gcc --print-prog-name=mipsel-openwrt-linux-gcc
mipsel-openwrt-linux-gcc
root@e95d6cbf1fcf:/build# mips-openwrt-linux-gcc --print-prog-name=mips-openwrt-linux-gcc
mips-openwrt-linux-gcc
root@e95d6cbf1fcf:/build# mips-openwrt-linux-gcc --print-prog-name=gcc
gcc

they do, however, have unique -print-search-dirs prints!

Making it conditional on a heuristic that checks whether the compiler binary actually starts with a shebang might be good enough?

Yeah that sounds totally plausible to me, and that way if someone wants to skip the extra invocation you could just switch it with a native executable (or something similar)

[alex]

Fun variant on this: on macOS, the clang binary is actually a weird find-the-real-clang indirection thing:

~ ❯❯❯ nm /usr/bin/clang
0000000100000000 T __mh_execute_header
0000000100000f77 T _main
0000000100001018 S _shim_marker
                 U _xcselect_invoke_xcrun
                 U dyld_stub_binder

The last modified time on /usr/bin/clang for me is the 4th of July, which is interesting because a) I'm quite certain I wasn't using this computer on the 4th, b) I just upgraded my clang this morning. From this I conclude, that for all practical purposes, the "compiler changed" logic is never invoked for the system clang on macOS.

(For some additional color... I discovered this because my Firefox builds are done with ASAN. On macOS, ASAN has a symbol with a version in it, presumably to prevent incompatibility between the compiler-emitted ASAN bits and the runtime library. The result is that if you try to use the new linker with an old object file, you get an error as a result. This is not a part of upstream ASAN AFAIK.)

[glandium]

Random though after having hit a similar problem again, how about hashing the output of $compiler --version instead of the binary?