Use LMDB as a cache backend #5509
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request has been linked to and will mark 1 task as "Done" when merged:
|
Benchmark ResultsKitchen Sink ✅
Timings
Cold Bundles
Cached Bundles
React HackerNews ✅
Timings
Cold Bundles
Cached Bundles
AtlasKit Editor ✅
Timings
Cold Bundles
Cached Bundles
Three.js ✅
Timings
Cold BundlesNo bundle changes detected. Cached Bundles
|
mischnic
reviewed
Dec 21, 2020
mischnic
reviewed
Dec 21, 2020
# Conflicts: # packages/core/core/src/requests/AssetGraphRequest.js # yarn.lock
Causes serializer errors otherwise
Co-authored-by: Niklas Mischkulnig <[email protected]>
|
I tried this out on a large app, and got this after a few transformations: |
wbinnssmith
force-pushed
the
lmdb-cache
branch
2 times, most recently
from
c2f6ac5 to
4b4ce35
Compare
# Conflicts: # packages/core/cache/package.json # packages/core/cache/src/Cache.js # yarn.lock
mischnic
reviewed
May 28, 2021
| ), | ||
| ); | ||
|
|
||
| // TODO: don't replace hash references in binary files?? |
There was a problem hiding this comment.
I guess this wasn't introduced by your changes?
There was a problem hiding this comment.
Yeah just a problem I noticed. But not sure how to know which files we should do this for.
There was a problem hiding this comment.
We use this in the data url optimizer, but its detection logic is rather sketchy : gjtorikian/isBinaryFile#22
mischnic
approved these changes
May 28, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements support for multiple cache backends, and adds an LMDB based implementation. LMDB is a very fast embedded key/value store written in C. Rather than write each cache entry to the file system separately, it uses one large memory-mapped file which is asynchronously written to disk. Overall this results in ~15% faster builds, and is even faster than using our own MemoryFS instead of NodeFS.
This PR automatically enables the LMDB-based cache when using a NodeFS, otherwise it falls back to the existing FS-based cache implementation.
Close T-265