Conversation
|
Updated 7:00 AM PT - Sep 25th, 2026
✅ @robobun, your commit 598cd9d8cdc7132d9bf828721e06fba97e49efb9 passed in 🧪 To try this PR locally: bunx bun-pr 43910That installs a local version of the PR into your bun-43910 --bun |
|
Status: draft. The base of this PR is the branch of #43897. Two commits in the diff are copies of #43871. The merge order and the open sibling PRs are in the Notes of the description. Reproduced on canary 1.4.3+367d939d9 and on a debug build of main (73df7bb), Linux x64: const f = Bun.file("/proc/self/status");
f.size; // or: await f.exists()
await f.text(); // "" (a fresh Bun.file(p).text() returns about 1500 bytes)
const out = Bun.file("out.txt"); // does not exist yet
await out.exists();
await Bun.write(out, Bun.file("in.txt")); // returns 0, out.txt is empty (#4930)
const g = Bun.file("four-bytes.txt");
await g.exists();
await Bun.write(g, "asdfasdf");
await g.text(); // "asdf" (#23902)All three return the right bytes on this branch. The PR has 67 new tests in This revision is narrower than the first one. |
ReadFile skipped the read when its fstat said st_size 0 and the store's cached mode said regular file. That cache is only filled once something stats the shared store, which clone() does since #42053 through store_reads_repeatably. new Response(Bun.file('/proc/version')).clone() then made both copies and the Bun.file itself read as "": a procfs file is a regular file with st_size 0 and real content. Drop the shortcut and read to EOF. A truly empty file costs one read() that returns 0.
2151da9 to
0af1354
Compare
…or a copy
`Blob::resolve_size` copied the cached `st_size` of a file onto the blob.
Readers and the file copy of `Bun.write` used it as a byte budget. A
procfs file reports 0 bytes and has content, and a file can grow.
- `resolve_size` stores no size on a file view. A file view keeps
`MAX_SIZE` ("until EOF") unless `slice()` set a window. `.size`,
`expect()` and the HEAD `Content-Length` take the length from
`resolved_size()`.
- After a stat, `ReadFile`, `ReadFileUV` and `FileReader` stop at the
size from their own `fstat`, so a read after `.size` needs no `read()`
at EOF. On Windows the stream reader asks `GetFileSizeEx`.
- `slice()` on a handle that nothing statted behaves as before and runs
no stat. After a stat, an index from the end counts from the length of
the stat, and `.size` of a window stops at that length.
- `do_sendfile` clamps the window against its own `fstat` before it
decides if the response is a range.
0af1354 to
b214680
Compare
After .size or exists() on the handle, main sends Content-Length: 200000 and 100000 body bytes.
Fixes #4930, fixes #23902
Problem
f.sizeorawait f.exists(), a procfsBun.filereads""and uploads 0 bytes.Bun.write(dest, Bun.file(src))writes 0 bytes afterawait dest.exists().text()afterwrite()is stale.Blob::resolve_size(src/runtime/webcore/Blob.rs) copied the cachedst_sizeonto the blob. Readers andBun.writeused it as a budget.Fix
resolve_sizestores no size on a file view, which keepsMAX_SIZE("until EOF") unlessslice()set a window. Lengths come fromresolved_size().stat, a reader stops at its ownfstatsize, with no addedread().Background
Blobis a view (offset,size) onto a store.MAX_SIZEmeans unknown. A file store caches onestat.Blobcache and truncation inBun.write#25702, Honor Bun.file().slice() bounds in Bun.write and spawn stdin #41209). With nostatsize on the blob, none is needed..text()does not return correct content after BunFile.write()#23902). A reader'sfstatsaves the sameread().Downsides
.sizeno longer bounds the read after it. After a growth from 4 to 8 bytes,text()returns 8 (was 4)..lastModified, a stream stops at the file size at open. Bytes appended during the read are not read (were)..sizeof a file takes 9.2 ns (was 4.8)..sizeandexists()on a missing file run onestatper call (was one).Notes
Stack and siblings
Merge order: #43871, then #43897, then this PR. The base of this PR is the branch of #43897. The diff also shows two commits that are copies of #43871, because this PR needs its change in
read_file.rs. They drop out in the rebase after #43871 merges.Open PRs that touch the same mechanism, and what this PR means for each:
resolve_size_for_stream. It keeps the cached size as the limit when it is not 0, this PR does not. A maintainer must choose. See the question belowresolve_size. This PR leaves S3 as on mainslice()index on a handle that nothing statted. This PR leaves that case as on mainFormDatapart, afetchupload and a module load of a file whosest_sizeis 0Fixes #4930. This PR fixes the destination half of it with no flag. Its source window and spawn stdin parts are not hereFixes #23902andFixes #22484. This PR fixes #23902. It does not fix #22484resolve_size()on a file blob in its creator. After this PR that call stores no size on the blob. The second to merge must check itQuestion for the reviewer of #41593: the cached size as the limit when it is not 0, or the size from the
fstatof the reader? Both need the same number ofread()calls (table below). The cached size truncates a file that grew after thestat(#23902).Behaviour, main against this PR
10-byte file
"0123456789"unless stated. "fresh" is aBun.filethat nothing statted. Release builds, Linux x64..sizeorexists(), thentext()""exists(), thenslice(0, 10).text()"""Linux vers"await dest.exists()on a missing file, thenBun.write(dest, Bun.file(src))dest.sizeon a 3-byte file, then the same copy"012"Bun.write(Bun.file(fd), src)after.size, 3-byte destinationexists(),Bun.write(f, "asdfasdf")on a 4-byte file,f.text()"asdf""asdfasdf".size, append 10 bytes,text()lengthS3Client.write(key, f),f.sizebefore the file grew from 10 to 20Bun.stdin.size, thenBun.stdin.text(), a file on stdin that grew from 10 to 20.sizebefore the file grew from 4 to 8Content-Length: 4Content-Length: 8Bun.file(fd),.sizebefore the file grew from 100000 to 200000 bytesContent-Length: 200000, 100000 body bytesContent-Length: 200000, 200000 body bytesBun.file(p).slice(5, 15)Content-Length: 10, 5 body bytesContent-Length: 5,bytes 5-9/*f.size, thenf.slice(5, 15)Content-Length: 5f.slice(-3).text()/f.slice(5, 5000).size/f.slice(0, 5).exists()"012"/ 4995 /falsef.size:slice(-3).text()/slice(5, 5000).size/slice(0, 5).exists()"789"/ 5 /truef.slice(-3), then append 10 bytes:f.sizeBun.write(f, "NEW"), thenprocess.exit(0), afterf.slice(-3): runs of 10 where the file isNEWA
Rangerequest past the end of a file that shrank answers 416 for a statted handle, as it does for a fresh handle. Main answered 200 with the whole file.slice()on a fresh handle runs nostatand is the same as on main, wrong results included. After astat, a window is not clamped whenslice()makes it..sizeclamps it against the cached length, a reader stops at EOF, anddo_sendfileclamps it against its ownfstat.Changed here, still not right
.sizekeeps the value of the firststat, and a read returns the file as it is now. So.sizecan be below the bytes read. On main both were the old length.stat, and GET reports the current length. On main both reported the old length. Bun.serve: resolve a file body for HEAD the same way as for GET #41585 owns HEAD.FormDatapart or afetchupload of a file over 256 KiB whosest_sizeis 0 stops at 262160 bytes, for a fresh handle and after astat. On main it was empty after astat. fetch: read a Bun.file() body with st_size 0 to EOF, up to the JS size limit #43964 owns the cut..sizeafterBun.write(f, ...)through the same handle is the old or the new length. It depends on whether.lastModifiedran between them. On main it is the old length in both orders.Bun.writeresets 1 of the 4 cachedstatfields.Unchanged from main
Bun.servesends a procfs file as an empty body withContent-Length: 0.exists()staystrueafter the file is deleted (await BunFile.exists()does not change #22484).slice()index on a fresh handle, and on a file whosest_sizeis 0..sizeandexists()of a slice of a fresh handle: the raw window andfalse.Bun.write(f, "NEW"), thenprocess.exit(0), afterf.sizeloses the write: 0 of 10 runs on main and here.st_size. This PR changes no reader of a pipe. No macOS machine ran a test for it.Measurements
Release builds of main (73df7bb) and of this PR, Linux x64.
perf,valgrind,hyperfine,straceandbloatyare not on the test machine. So the units are bytes (size,nm -S), syscalls (/proc/self/io,gdbwithcatch syscall) and wall-clock time as interleaved runs with an A-against-A noise floor. The host had a load average of 90 to 125..text, bytes (size)nm -S).size, statted file, ns per access.size, file slice, ns per access.size, in-memoryBlob, ns per access.size, missing file, ns per accessstatper access.bodydrain, 64 KiB, µs.bodydrain, 256 KiB, µsBlobhas no new field (12 fields on both sides).store::Filegets one method,is_statted(). The structured-clone wire version is 4 on both sides.readsyscalls per operation, 64 KiB file,/proc/self/io(syscr) over 300 iterations:text().size, thentext()exists(), thentext().lastModified, thentext()stream().size, thenstream().lastModified, thenstream()new Response(file).bodyStat-family syscalls for 50 calls (
gdb,catch syscall, process startup taken off):Bun.file(p).size50 and 50.f.sizeandawait f.exists()on one handle 1 and 1.f.sizeandawait f.exists()on a missing file 1 and 50. Eachslice()case (slice(0, 10).size,slice(-5),slice(5),slice.exists(), fresh or one handle) 0 and 0.Windows x64, I/O operations per operation from the
Win32_Processcounters, 64 KiB file, 300 iterations. "read" isReadOperationCount. "other" isOtherOperationCount. Main is a release build and this PR is a debug build.text().size, thentext()stream().size, thenstream().lastModified, thenstream()new Response(file).bodyThe stream reader on Windows runs no
fstatat open. It asks the open handle for the size withGetFileSizeEx, and only when the earlierstatfound a regular file. A variant withfstatgave 8.18 other operations in the last three rows.Self-review
Two rounds. The second round raised 29 concerns on the first revision of this PR. 25 are addressed:
slice(-n),.sizeof a slice andexists()of a slice cached astaton the store that the parent handle shares. After them the parent's.sizedid not see an append,exists()did not see a deletion, andBun.write(f, "NEW")beforeprocess.exit(0)lost the write. They run nostatnow, and 5 guard tests pin that.MAX_SIZE - n. It counts from 0 now.do_sendfilesentContent-Length: 10with 5 body bytes forf.size, thenf.slice(5, 15). It clamps first now.serve.test.ts. Bun.file().slice(): count a negative index back from the file's real end #41257 owns it.Bun.file(fd), an S3 upload of a statted file,Bun.file(fd)as source and as destination, a file on stdin, and one matrix of 5 stat calls, 4 changes and 6 readers.Open:
statonstore::Filehas no single owner: 4 fields, 5 writers, andBun.writeresets 1 field. That is a PR of its own after this one.resolve_size()keeps its name while a file view no longer gets a size from it. A rename is a PR with no behaviour change.Suites
Debug build, Linux x64, on the pushed source:
bun-file-read41 pass.bun-file-fd-read6.bun-stdin-slice10.bun-serve-file136.body776.body-clone86.blob110.wasm-streaming33.structured-clone-blob-file42.expect416.streams624.fetch-file-upload11.bun-file6.bun-file-exists1.FormData149. TheContent-Rangetests ofserve.test.ts45.bun-write92 to 96 pass: 1 to 5 tests that spawn a child go over their 5 s limit on this machine, with main'ssrc/too.fetch.test.ts374 pass and 22 fail: 21 need IPv6, the public internet or a non-root user, and the 22nd is a memory bound of 256 MB that measures 258 to 265 MB. All 22 fail with main'ssrc/too. The wholeserve.test.ts329 pass and 2 fail: a root port and/bun:info, which fail with main'ssrc/too.Windows x64, debug build, exit codes checked:
bun-file-read27 pass.bun-serve-file128.body774.body-clone84.blob108.streams617.bun-file6.bun-file-exists1.FormData149.fetch-file-upload11.structured-clone-blob-file38.wasm-streaming33.expect416.bun-stdin-slice5. TheContent-Rangetests ofserve.test.ts45. Exit code 0 for each file.bun-write85 pass and 1 fail:Bun.write() without uv_fs_copyfile, which fails on main there too.bun run rust:check-allpasses on 12 targets.no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/web/fetch/body.test.ts, test/js/web/fetch/body-clone.test.ts, test/js/bun/util/bun-stdin-slice.test.ts, test/js/bun/util/bun-file-read.test.ts, test/js/bun/util/bun-file-fd-read.test.ts, test/js/bun/io/bun-write.test.js, test/js/bun/http/serve.test.ts, test/js/bun/http/bun-serve-file.test.ts