-
-
Notifications
You must be signed in to change notification settings - Fork 9.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
go: support & use aggressive caching #4774
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I hadn't planned to move on this so quickly but impressively core today had its first Go Module using PR, so this has become more useful than it would've been. Useful reading on the Modules system can be found via https://golang.org/doc/go1.11#modules. `GOCACHE` has been a thing for a while but it hasn't really been especially beneficial to us, so I've ignored it, but with the Go Modules system it can be incredibly useful in reducing build times & thus CI burden. A lot of this is explained rather nicely here: https://groups.google.com/forum/#!msg/golang-dev/RjSj4bGSmsw/KMHhU8fmAwAJ The upsides here for us are obvious: 1) We do a *lot* of go-based builds, and the more things move over to the module system the more comprehensive that cache will become and we'll need to remotely fetch less dependencies, which massively speeds up build times even on high-speed connections. 2) Module dependencies aren't limited per formula. If `hugo` fetches a dependency that is the same version as, for example, `wiki` also needs `wiki` won't bother to fetch that version remotely but instead will simply pluck it out of the cache. If we look at `hugo`, because it's the only formula so far with Module support: First build: `built in 1 minute 41 seconds` Second build: `built in 14 seconds`. It's worth noting perhaps that we can use just the module cache rather than the build cache + module cache combination, but doing so leads to a significant reduction in regained build time. A build under that system pops an average build time with `hugo` (after the first build) of ~70 seconds, which is only a 30 second saving on the first build. However, if we wanted to speed up builds whilst also not eating big chunks of disk space that is an option. The downside of a build cache + module cache system is disk usage. `go` does not exactly eat up disk space sparingly, and the cache size from `hugo` alone is 258.8MB. That could be an issue on CIs where disk space is already a squeeze at times, but it's not much different to how we let the `java_cache` behave now. All Module-supporting formulae need to do to use the shared cache is instead of deleting the `GOPATH` env we set everywhere replace it with this: ``` ENV["GOPATH"] = "#{HOMEBREW_CACHE}/go_cache" ```
MikeMcQuaid
approved these changes
Aug 29, 2018
Nice work here @DomT4 👍 |
4 tasks
Thanks Mike! I'll try to keep an eye on how much this whacks CI on disk usage and review again if necessary. There's a follow-up in #4778 to resolve an issue I missed initially, le sigh. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
brew style
with your changes locally?brew tests
with your changes locally?I hadn't planned to move on this so quickly but impressively core today had its first Go Module using PR, so this has become more useful than it would've been. Useful reading on the Modules system can be found via https://golang.org/doc/go1.11#modules.
GOCACHE
has been a thing for a while but it hasn't really been especially beneficial to us, so I've ignored it, but with the Go Modules system it can be incredibly useful in reducing build times & thus CI burden. A lot of this is explained rather nicely here.The upsides here for us are obvious:
hugo
fetches a dependency that is the same version as, for example,wiki
also needswiki
won't bother to fetch that version remotely but instead will simply pluck it out of the cache.If we look at
hugo
, because it's the only formula so far with Module support:First build:
built in 1 minute 41 seconds
Second build:
built in 14 seconds
.It's worth noting perhaps that we can use just the module cache rather than the build cache + module cache combination, but doing so leads to a significant reduction in regained build time. A build under that system pops an average build time with
hugo
(after the first build) of ~70 seconds, which is only a 30 second saving on the first build. However, if we wanted to speed up builds whilst also not eating big chunks of disk space that is an option.The downside of a build cache + module cache system is disk usage.
go
does not exactly eat up disk space sparingly, and the cache size fromhugo
alone is 258.8MB. That could be an issue on CIs where disk space is already a squeeze at times, but it's not much different to how we let thejava_cache
behave now. The major difference betweenjava_cache
andgo_cache
here I guess is that things that usego
are much more regular visitors in the CI system.All Module-supporting formulae need to do to use the shared cache is instead of deleting the
GOPATH
env we set everywhere replace it with this: