Optimize GitHub Actions cache usage #6555
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The Go cache has fixed trim settings. It updates mtimes hourly and trims daily, deleting anything older than five days. This is too conservative for CI caches. We only want to retain caches used in the current run. We can achieve this by setting the mtime of all cache files to a time between one hour and five days ago. After the build, all used cache entries will have a newer mtime. Everything else can be trimmed. Signed-off-by: Tom Wieczorek <[email protected]>
It only provides an approximate 20-second speed boost, while eating up a couple of 100MB per GitHub cache. We can add it back later on if we think it's really worth it. Signed-off-by: Tom Wieczorek <[email protected]>
twz123
force-pushed
the
optimize-actions-caches
branch
from
0b62d46 to
304dd78
Compare
Simplify the caching of the embedded executables. Makes the cache size smaller, at the cost of some extra packaging time spent to build the gzipped blob during build time. This requires some tricks to tell make what needs to be built and what doesn't. Only keep the most recent cache per branch. All caches ending in a hash will be pruned so that only the newest cache is retained on a per-branch and per-key-prefix basis. Don't use leading slashes for URLs passed to the gh CLI. Leading slashes would otherwise trigger file path normalization in bash on Windows, which would render the URL fragment illegal. Use a fixed source date epoch in unit tests to better leverage cache usage. Signed-off-by: Tom Wieczorek <[email protected]>
twz123
force-pushed
the
optimize-actions-caches
branch
from
304dd78 to
8eb3ccd
Compare
twz123
added
area/ci
chore
backport/release-1.34
Contributor
|
Successfully created backport PR for |
16 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The Go cache has fixed trim settings. It updates mtimes hourly and trims daily, deleting anything older than five days. This is too conservative for CI caches. We only want to retain caches used in the current run. We can achieve this by setting the mtime of all cache files to a time between one hour and five days ago. After the build, all used cache entries will have a newer mtime. Everything else can be trimmed.
Remove the caching of the Go module cache. It only provides an approximate 20-second speed boost, while eating up a couple of 100MB per GitHub cache. We can add it back later on if we think it's really worth it.
Simplify the caching of the embedded executables. Makes the cache size smaller, at the cost of some extra packaging time spent to build the gzipped blob during build time. This requires some tricks to tell make what needs to be built and what doesn't.
Only keep the most recent cache per branch. All caches ending in a hash will be pruned so that only the newest cache is retained on a per-branch and per-key-prefix basis.
Don't use leading slashes for URLs passed to the gh CLI. Leading slashes would otherwise trigger file path normalization in bash on Windows, which would render the URL fragment illegal.
Use a fixed source date epoch in unit tests to better leverage cache usage.
Trying to mitigate this:
Type of change
How Has This Been Tested?
Checklist