Skip to content

Commit

Permalink
Run cargo clean on CI when the version of rustc changes
Browse files Browse the repository at this point in the history
Recently I've been monitoring where our CI time is spent.  Currently,
the build and test stages take about 14 minutes without a cache and
1.7 to 3.6 minutes when a cache is present.  The variation in the
latter comes from the `cargo install diesel_cli` step which is
partially rebuilt any time a new compatible version of a dependency
is published.

For each channel, we cache about 1GB of build artifacts.  This results
in a total of 4 minutes that are spent in caching related activities.
However, extracting the cache on Travis appears to be O(n^3) because
once a new version of rustc is released the size of the `target/`
directory doubles but cache extraction take 8 times as long and total
cache related time nearly quadruples to 13.5 minutes.  This effectively
wipes out any gains of caching build artifacts for our codebase,
especially on the beta and nightly channels.

This commit adds a shell script that is run before any build steps on
CI.  The script works by recording the version of rustc in a file under
`target/`.  Upon the next build, if the cache was created by the same
version of rustc then no action is taken.  However, if the version of
rustc has changed then `cargo clean` is run to clear out the stale
cached files.

This commit also sets the cache timeout (for  building and uploading
the cache) to 6 minutes.  The default is 3 minutes and this portion is
currently taking about 2.6 minutes.  The rationale here is to provide
some additional headroom on the stable channel.  As dependencies are
updated, the cache will gradually grow until a new release causes the
cache to be cleared.  The increased timeout should allow for this
natural growth while also enforcing a reasonable limit so that caching
overhead doesn’t grow too large.
  • Loading branch information
jtgeibel committed Dec 14, 2018
1 parent 614b33c commit 453b057
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 4 deletions.
7 changes: 3 additions & 4 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ cache:
cargo: true
directories:
- $HOME/.npm
timeout: 600
# Set timeout to 6 minutes (double the default of 3 minutes)
timeout: 360

env:
global:
Expand All @@ -26,6 +27,7 @@ env:
- PERCY_PROJECT=crates-io/crates.io

install:
- script/cargo-clean-on-new-rustc-version.sh
- cargo install --force diesel_cli --vers `cat .diesel_version` --no-default-features --features postgres && export PATH=$HOME/.cargo/bin:$PATH

before_script:
Expand Down Expand Up @@ -68,9 +70,6 @@ matrix:
script:
- cargo build
- cargo test
# This portion of the cache is quickly invalidated anyway
before_cache:
- cargo clean

notifications:
email:
Expand Down
22 changes: 22 additions & 0 deletions script/cargo-clean-on-new-rustc-version.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/bin/sh

set -e

stamp_file=target/rustc_version_stamp
current_version=$(rustc --version)

if [ -f $stamp_file ]; then
# Compare the current version against the previous version
if echo "$current_version" | cmp -s $stamp_file -; then
echo "Version of rustc hasn't changed, keeping the cache intact"
else
echo "The version of rustc has changed, running cargo clean"
cargo clean
fi
else
echo "There is no existing version stamp, keeping the cache intact"
fi

# Save the version stamp for next time
mkdir target/
echo $current_version > $stamp_file

0 comments on commit 453b057

Please sign in to comment.