Add (glob), (only-globs), and (except-globs) for fine-grained caching via include/exclude glob patterns
#282
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
+ fix up tests + pend glob test since it fails. seems like this only works for local dirs. :(
Using labels in this way was always a bit confusing, since labels would otherwise be appropriate to set on container images, where they don't normally bust caches. Use env vars instead. On the plus side, now globs don't bust caches, which is much more useful than using labels to! (Classic carrot + stick scenario.)
* push file vs directory logic down into Includes()/Excludes() * file globbables implicitly include only the specified file
they don't actually prevent cache busting at the moment, for the same reason it was broken in Bass (setting hostname busts cache)
timestamp wasn't enough to prevent concurrent runs of same tests via gRPC->Buildkit + Buildkit
this was always a decision of convenience, doesn't make sense when they're more useful as image metadata, which shouldn't bust caches
tricky interaction between thunk output caching + cache dirs. now that thunk names don't cause a cache bust for "free" we need to more explicitly bust them so the next test run doesn't just see the output generated later in the test run.
is this the same as before? maybe caches themselves aren't a part of the outer cache key?
Suspecting this is the root cause of duplicate runs in cache test, but my understanding of it is hazy at best. Seems like this should have been cached regardless.
This reverts commit 576d9ab.
This reverts commit 7ccae86.
vito
changed the title
(glob) for fine-grained caching via include/exclude filters(glob), (only-globs), and (except-globs) for fine-grained caching via include/exclude glob patterns
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a major new feature for fine-grained cache control.
Now you can take a thunk path or a host path and use
(glob)to scope it down to certain paths:❯ bass => (glob *dir* ./**/*.go ./Makefile) <host: .>/?[./**/*.go ./Makefile]When passing this path to another thunk, not only will only the requested files be included, and downstream consumers will only have their cache busted if the requested files changed.
You can also exclude files by adding a
!argument ahead of them:=> (glob *dir* ./**/*.go ./Makefile ! ./main.go) <host: .>/?[./**/*.go ./Makefile]![./main.go]Excludes are processed after includes, so the above example will include all
*.gofiles, the aMakefile, but will not includemain.go.Check out the changes to
bass/bass.bassin this PR for real-world usage examples. Notably this completely removed the need for(subgit)which was always janky; it was git-specific so it wouldn't play nice if you were running from local paths.Breaking change
thunkhostname instead of the thunk name.(with-label)no longer busts caches. (This used to work by virtue of changing the hostname.)This change was necessary because otherwise thunk caches would bust when they run against new inputs (e.g.
git clonefor a new commit) even if the glob rules scoped the changes out.Having
(with-label)bust caches became really awkward when I wanted to use them for OCI image labels, so at least that change was probably inevitable.The docs have been updated to use
(with-env)as the cache buster instead, since this directly affects the LLB.