Write cache large blobs in chunks to avoid write file limits in Node #9355
Conversation
JakeLane
requested review from
lettertwo,
mattcompiles,
gorakong,
thebriando and
AGawrys
packages/dev/query/src/index.js
Outdated
| @@ -120,9 +124,9 @@ export function loadGraphs(cacheDir: string): {| | |||
| return {assetGraph, bundleGraph, requestTracker, bundleInfo}; | |||
| } | |||
|
|
|||
| function loadLargeBlobRequestRequestSync(cacheDir, node) { | |||
| async function loadLargeBlobRequestRequestSync(cache, node) { | |||
There was a problem hiding this comment.
I'm not sure why this was originally written to be sync, but if we're making it async then we should rename the function.
There was a problem hiding this comment.
If I recall correctly, I made loadGraphs sync just to make it easier to use from the Node REPL. So no big deal
packages/core/cache/src/types.js
Outdated
| @@ -12,7 +12,7 @@ export interface Cache { | |||
| setBlob(key: string, contents: Buffer | string): Promise<void>; | |||
| hasLargeBlob(key: string): Promise<boolean>; | |||
| getLargeBlob(key: string): Promise<Buffer>; | |||
| setLargeBlob(key: string, contents: Buffer | string): Promise<void>; | |||
There was a problem hiding this comment.
Not sure if this ever actually happens, but this is technically a breaking change for this package. Is it definitely required?
There was a problem hiding this comment.
It's not required but I figured it would be a nice clean up to do, since it's adding complexity. Is there a convention for tightening APIs like this?
I'll remove the API change anyway, since it's relatively easy to implement
packages/core/cache/src/LMDBCache.js
Outdated
| writePromises.push( | ||
| this.fs.writeFile( | ||
| this.#getFilePath(key, i), | ||
| contents.subarray(i * WRITE_LIMIT_CHUNK, (i + 1) * WRITE_LIMIT_CHUNK), |
There was a problem hiding this comment.
Might be a micro-optimization, but should we avoid this work when chunks.length === 1 seeing as that's most of the time?
There was a problem hiding this comment.
Yeah fair call, there's no copy cost as it references the same memory in the buffer, but there's no guarantee the work is trivial. Will change it when updating the rest of the code
JakeLane
force-pushed
the
jlane2/chunk-large-blogs-to-disk
branch
from
862e9fe
to
3a79744
Compare
↪️ Pull Request
In Node, there's a limit to the size we can read at once (2 GiB). We are hitting that for large monorepos at Atlassian. This makes the cache unusable for repeated usage, as the node API for readFile will always return an error.
A mitigation for this is to chunk the files in ~2GiB segments. This means the upper limit of blob size will be the buffer size limit.
On my machine, the buffer size limit is:
This gives us some headroom to resolve large blob size issues.