Make cache dependent on "env" OR allow dynamic customization of the cache key

[LionsAd]

The cache is great, but unfortunately for different environment variables the cache runs into a race condition.

Test case:

matrix:
  env:
    - JOB="A"
    - JOB="B"

• Job A: Waits 5 seconds and creates $CACHE_DIR/a.txt
• Job B: Waits 20 seconds Creates $CACHE_DIR/b.txt

Because Job A and B are running at the same time, but Job B takes longer, Job B wins.

So the cache now has:

$CACHE_DIR/b.txt

But then Job A overwrites the cache again.

With some luck it starts early and has the $CACHE_DIR/b.txt and then can add its a.txt, but because B runs afterwards it will still overwrite with only b.txt, etc.

That unfortunately limits the cache usage.

Is there any other thing I can do right now to use something else in the build matrix, which the cache key is dependent on?

[LionsAd]

Playing with this some more, the simplest would be to allow:

matrix:
  env:
    - JOB="A" TRAVIS_CACHE_KEY="A"
    - JOB="B" TRAVIS_CACHE_KEY="B"

[LionsAd]

Spoke with @joshk in IRC:

TRAVIS_CACHE_KEY makes things complicated, current best idea is to just md5() the whole env line and add that md5 / other hash key to the cache key.

[joshk]

Downside is that many times it is fine to reuse a cache between jobs, thus this would explode caching.

Maybe this is something which could be turned on/off.

[LionsAd]

Yes, agree. Cache sharing can make sense. (hence my proposal with TRAVIS_CACHE_KEY), but I am fine with a on/off switch.

Just would like to make it possible to have some more granularity.

[coderanger]

Another use case for this, it would be nice to be able to have a single Gemfile that checks/uses an env var to control the version of a single gem rather than making a handful of actual Gemfiles. Currently I have to do the latter or the bundler cache thrashes back and forth.

[cvrebert]

Duplicate of #4393?

[BanzaiMan]

Just a heads up: I deployed a fix to production about 10 minutes ago. Please observe how your caches are behaving, and report issues if you see any.

[BanzaiMan]

This has been deployed and documented.

[ctubbsii]

@BanzaiMan Do you have a link to the documentation?

[BanzaiMan]

https://docs.travis-ci.com/user/caching/#Caches-and-build-matrices

[ctubbsii]

From the docs, it looks like they explain how to use an env variable to use distinct caches for different tasks in a matrix build. However, there still does not appear to be a way to share caches across jobs (to make PRs reuse cached maven repository from previous PRs, for example).