Correct dist-toolchain cache invalidation

[chmanchester]

A user reported needing to delete the dist-client cache directory locally when moving between rust toolchain versions to correct a version mismatch between local and the distributed builder. We should determine the circumstances where this might happen and correct it.

[luser]

The dist client has this concept of "weak toolchain hashes" which are an easy-to-compute hash of some parts of the toolchain:

sccache/src/dist/cache.rs

Lines 47 to 55 in d309339

	// Local machine mapping from 'weak' hashes to strong toolchain hashes
	// - Weak hashes are what sccache uses to determine if a compiler has changed
	// on the local machine - they're fast and 'good enough' (assuming we trust
	// the local machine), but not safe if other users can update the cache.
	// - Strong hashes (or 'archive ids') are the hash of the complete compiler contents that
	// will be sent over the wire for use in distributed compilation - it is assumed
	// that if two of them match, the contents of a compiler archive cannot
	// have been tampered with
	weak_map: Mutex<HashMap<String, String>>,

It uses those to lookup whether it has already packaged a toolchain, and during packaging it generates a stronger hash from all the files in the toolchain archive, which is what it uses to talk about toolchains with the build server.

For the Rust compiler the weak hash is actually pretty good, it includes the compiler_shlibs_digests that get calculated when sccache finds a Rust compiler (the hashes of all the shared libraries in $(rustc --print=sysroot)/lib, or /bin on Windows):

sccache/src/compiler/rust.rs

Lines 1003 to 1011 in 6371211

	let mut m = Digest::new();
	// Hash inputs:
	// 1. A version
	m.update(CACHE_VERSION);
	// 2. compiler_shlibs_digests
	for d in compiler_shlibs_digests {
	m.update(d.as_bytes());
	}
	let weak_toolchain_key = m.clone().finish();

The weak map gets persisted as ~/.cache/sccache-dist-client/client/weak_map.json on Linux so it should be possible to have someone send you that file if they hit this again.

It's also possible that this is related to #87 will not do the right thing if someone uses rustup to point the same rustc binary at a different actual version of rustc during the lifetime of a server process, since the server caches compiler info purely based on path + mtime:

sccache/src/server.rs

Line 839 in d309339

//TODO: properly handle rustup overrides. Currently this will

[chmanchester]

This seems like #87

[chmanchester]

On further inspection, there's the override case to handle, but also the simpler case of a binary path like .rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc getting an update that we fail to detect. Re-opening this issue to handle the latter.

[luser]

That seems like it ought to be handled properly? The sccache server caches compiler info by path + mtime, so if a compiler gets updated that ought to invalidate the cache.