rust cache on circleci keeps growing

[ericseppanen]

One of the longest steps in a rust build is saving and restoring the rust cache. I noticed today that the size of that cache had suddenly jumped from 1.3GB to 2.8GB.

I think what's happening here is that Cargo keeps every version of every dependency you've ever built in target/, and in ~/.cargo.

This really doesn't play nicely with the style of caching that circleci encourages: try to recover a perfect dependency cache, and then fall back on an imperfect match. Repopulate anything missing into the same directory structure, and then store the result as the new cache entry. This results in unbounded growth over time as historical packages accumulate.

[ericseppanen]

I'm not sure what the right strategy should be for pruning garbage (outdated packages) from the cache, but since I managed to increase the size by 1.5GB in one day, drastic action is needed.

I pushed a commit that flushes the cache (by modifying the cache key), and disables the "imperfect match" on the cache. This will increase build time a lot on any commit that modifies Cargo.lock, but at least buys us some time to find a more sustainable strategy.

[ericseppanen]

cargo-cache has some tools for managing the size of ~/.cargo, as well as some good explanations of the various subdirectories, and which can be safely deleted.

There is some good discussion about managing target/ here, but there doesn't seem to be a consensus about how to manage it effectively.

[ericseppanen]

After researching this more, I don't think we should do anything more about this right now. We have a lot of dependencies. Compiled, they take 4.4GB of space, and we're lucky they compress down to only 1.3GB. It takes time to load and save these compiled dependencies, so we're just going to have to pay the cost of 1-2 minutes per build job.

I think the "emergency" fix (rebuilding everything from scratch whenever Cargo.lock changes) will just have to be the permanent fix, at least for now.