Skip to content

Bun.file: do not use a cached stat size as the byte budget of a read or a copy - #43910

Draft
robobun wants to merge 5 commits into
robobun/c897cd79/blob-resolve-size-one-implfrom
robobun/c897cd79/file-stat-not-a-budget
Draft

robobun wants to merge 5 commits into
robobun/c897cd79/blob-resolve-size-one-implfrom
robobun/c897cd79/file-stat-not-a-budget

Conversation

@robobun

@robobun robobun commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4930, fixes #23902

Problem

  • After f.size or await f.exists(), a procfs Bun.file reads "" and uploads 0 bytes.
  • Bun.write(dest, Bun.file(src)) writes 0 bytes after await dest.exists(). text() after write() is stale.
  • Cause: Blob::resolve_size (src/runtime/webcore/Blob.rs) copied the cached st_size onto the blob. Readers and Bun.write used it as a budget.

Fix

  • resolve_size stores no size on a file view, which keeps MAX_SIZE ("until EOF") unless slice() set a window. Lengths come from resolved_size().
  • After a stat, a reader stops at its own fstat size, with no added read().
  • Verified: 67 new tests. A debug build of main fails 42, the rest are guards.
  • Self-reviewed: 29 concerns raised, 25 addressed, 4 open (Notes).

Background

Downsides

  • .size no longer bounds the read after it. After a growth from 4 to 8 bytes, text() returns 8 (was 4).
  • After .lastModified, a stream stops at the file size at open. Bytes appended during the read are not read (were).
  • .size of a file takes 9.2 ns (was 4.8). .size and exists() on a missing file run one stat per call (was one).
Notes

Stack and siblings

Merge order: #43871, then #43897, then this PR. The base of this PR is the branch of #43897. The diff also shows two commits that are copies of #43871, because this PR needs its change in read_file.rs. They drop out in the rebase after #43871 merges.

Open PRs that touch the same mechanism, and what this PR means for each:

PR Relation
#43871 Needed. Copied here until it merges
#43897 Needed. The base of this PR
#41593 Its 7 procfs tests pass here with no resolve_size_for_stream. It keeps the cached size as the limit when it is not 0, this PR does not. A maintainer must choose. See the question below
#43934 Owns the S3 arm of resolve_size. This PR leaves S3 as on main
#41257 Owns a negative slice() index on a handle that nothing statted. This PR leaves that case as on main
#41585 Owns HEAD for a file body. This PR changes 2 lines of the same arm, so the second to merge rebases
#41590, #43964, #43965 Own the 256 KiB cut of a FormData part, a fetch upload and a module load of a file whose st_size is 0
#41209 Also says Fixes #4930. This PR fixes the destination half of it with no flag. Its source window and spawn stdin parts are not here
#39763 Also says Fixes #23902 and Fixes #22484. This PR fixes #23902. It does not fix #22484
#43896 Calls resolve_size() on a file blob in its creator. After this PR that call stores no size on the blob. The second to merge must check it
#43916, #40221 Touch the same files

Question for the reviewer of #41593: the cached size as the limit when it is not 0, or the size from the fstat of the reader? Both need the same number of read() calls (table below). The cached size truncates a file that grew after the stat (#23902).

Behaviour, main against this PR

10-byte file "0123456789" unless stated. "fresh" is a Bun.file that nothing statted. Release builds, Linux x64.

case main this PR
procfs: .size or exists(), then text() "" the content
procfs: exists(), then slice(0, 10).text() "" "Linux vers"
await dest.exists() on a missing file, then Bun.write(dest, Bun.file(src)) writes 0 writes 10
dest.size on a 3-byte file, then the same copy 3, "012" 10
Bun.write(Bun.file(fd), src) after .size, 3-byte destination 3 10
exists(), Bun.write(f, "asdfasdf") on a 4-byte file, f.text() "asdf" "asdfasdf"
.size, append 10 bytes, text() length 10 20
S3Client.write(key, f), f.size before the file grew from 10 to 20 PUT of 10 bytes PUT of 20 bytes
Bun.stdin.size, then Bun.stdin.text(), a file on stdin that grew from 10 to 20 10 bytes 20 bytes
Bun.serve GET of one reused handle, .size before the file grew from 4 to 8 206, Content-Length: 4 200, Content-Length: 8
Bun.serve GET of one open Bun.file(fd), .size before the file grew from 100000 to 200000 bytes Content-Length: 200000, 100000 body bytes Content-Length: 200000, 200000 body bytes
Bun.serve GET, fresh Bun.file(p).slice(5, 15) 200, Content-Length: 10, 5 body bytes 206, Content-Length: 5, bytes 5-9/*
Bun.serve GET, f.size, then f.slice(5, 15) 206, Content-Length: 5 the same
fresh f.slice(-3).text() / f.slice(5, 5000).size / f.slice(0, 5).exists() "012" / 4995 / false the same
after f.size: slice(-3).text() / slice(5, 5000).size / slice(0, 5).exists() "789" / 5 / true the same
fresh f.slice(-3), then append 10 bytes: f.size 20 20
Bun.write(f, "NEW"), then process.exit(0), after f.slice(-3): runs of 10 where the file is NEW 10 10

A Range request past the end of a file that shrank answers 416 for a statted handle, as it does for a fresh handle. Main answered 200 with the whole file.

slice() on a fresh handle runs no stat and is the same as on main, wrong results included. After a stat, a window is not clamped when slice() makes it. .size clamps it against the cached length, a reader stops at EOF, and do_sendfile clamps it against its own fstat.

Changed here, still not right

  • .size keeps the value of the first stat, and a read returns the file as it is now. So .size can be below the bytes read. On main both were the old length.
  • Bun.serve HEAD of a reused handle reports the length of the first stat, and GET reports the current length. On main both reported the old length. Bun.serve: resolve a file body for HEAD the same way as for GET #41585 owns HEAD.
  • A FormData part or a fetch upload of a file over 256 KiB whose st_size is 0 stops at 262160 bytes, for a fresh handle and after a stat. On main it was empty after a stat. fetch: read a Bun.file() body with st_size 0 to EOF, up to the JS size limit #43964 owns the cut.
  • .size after Bun.write(f, ...) through the same handle is the old or the new length. It depends on whether .lastModified ran between them. On main it is the old length in both orders. Bun.write resets 1 of the 4 cached stat fields.

Unchanged from main

  • Bun.serve sends a procfs file as an empty body with Content-Length: 0.
  • exists() stays true after the file is deleted (await BunFile.exists() does not change #22484).
  • A negative slice() index on a fresh handle, and on a file whose st_size is 0.
  • .size and exists() of a slice of a fresh handle: the raw window and false.
  • Bun.write(f, "NEW"), then process.exit(0), after f.size loses the write: 0 of 10 runs on main and here.
  • macOS reports the bytes that wait in a pipe as st_size. This PR changes no reader of a pipe. No macOS machine ran a test for it.

Measurements

Release builds of main (73df7bb) and of this PR, Linux x64. perf, valgrind, hyperfine, strace and bloaty are not on the test machine. So the units are bytes (size, nm -S), syscalls (/proc/self/io, gdb with catch syscall) and wall-clock time as interleaved runs with an A-against-A noise floor. The host had a load average of 90 to 125.

what main this PR note
release .text, bytes (size) 80,676,462 80,676,974 +512
function bytes (nm -S) net +386 36 functions changed, +2721 and -2335
.size, statted file, ns per access 4.83 9.15 paired delta +4.12, noise floor 0.81
.size, file slice, ns per access 5.28 6.69 paired delta +0.91, under the noise floor of 2.10
.size, in-memory Blob, ns per access 7.24 6.28 paired delta +0.85, under the noise floor of 1.88
.size, missing file, ns per access 10.0 558 one stat per access
.body drain, 64 KiB, µs 36.69 38.81 paired delta +0.72, under the noise floor of 3.97
.body drain, 256 KiB, µs 50.16 41.84 paired delta -5.63, noise floor 3.45

Blob has no new field (12 fields on both sides). store::File gets one method, is_statted(). The structured-clone wire version is 4 on both sides.

read syscalls per operation, 64 KiB file, /proc/self/io (syscr) over 300 iterations:

operation main this PR
fresh text() 2.03 2.03
.size, then text() 1.03 1.03
exists(), then text() 1.03 1.03
.lastModified, then text() 2.03 1.03
fresh stream() 2.10 2.10
.size, then stream() 1.08 1.08
.lastModified, then stream() 2.10 1.08
new Response(file).body 1.09 1.09

Stat-family syscalls for 50 calls (gdb, catch syscall, process startup taken off): Bun.file(p).size 50 and 50. f.size and await f.exists() on one handle 1 and 1. f.size and await f.exists() on a missing file 1 and 50. Each slice() case (slice(0, 10).size, slice(-5), slice(5), slice.exists(), fresh or one handle) 0 and 0.

Windows x64, I/O operations per operation from the Win32_Process counters, 64 KiB file, 300 iterations. "read" is ReadOperationCount. "other" is OtherOperationCount. Main is a release build and this PR is a debug build.

operation main, read main, other this PR, read this PR, other
fresh text() 2.01 4.17 2.01 4.17
.size, then text() 1.01 8.17 1.01 8.17
fresh stream() 2.01 2.14 2.01 2.15
.size, then stream() 1.01 6.17 1.01 6.18
.lastModified, then stream() 2.01 6.14 1.01 6.18
new Response(file).body 1.01 6.17 1.01 6.18

The stream reader on Windows runs no fstat at open. It asks the open handle for the size with GetFileSizeEx, and only when the earlier stat found a regular file. A variant with fstat gave 8.18 other operations in the last three rows.

Self-review

Two rounds. The second round raised 29 concerns on the first revision of this PR. 25 are addressed:

  • slice(-n), .size of a slice and exists() of a slice cached a stat on the store that the parent handle shares. After them the parent's .size did not see an append, exists() did not see a deletion, and Bun.write(f, "NEW") before process.exit(0) lost the write. They run no stat now, and 5 guard tests pin that.
  • A negative index on a statted empty file made a window at MAX_SIZE - n. It counts from 0 now.
  • do_sendfile sent Content-Length: 10 with 5 body bytes for f.size, then f.slice(5, 15). It clamps first now.
  • The S3 arm is out. s3: leave the size of an S3 blob unknown in resolve_size #43934 owns it.
  • The negative index on a fresh handle is out, with the change to serve.test.ts. Bun.file().slice(): count a negative index back from the file's real end #41257 owns it.
  • Two comments outside the diff stated the old contract. They are rewritten.
  • New tests: a file that grows during a stream, Bun.serve with a reused handle and with an open Bun.file(fd), an S3 upload of a statted file, Bun.file(fd) as source and as destination, a file on stdin, and one matrix of 5 stat calls, 4 changes and 6 readers.
  • The PR is restacked on Blob: resolve_size applies resolved_size #43897, and this body has the merge order and the siblings.
  • The title, the Downsides and the two lists above are corrected.

Open:

  1. No maintainer has chosen between this design and Leave a body stream unsized when the file's st_size is 0 #41593. The question is above.
  2. The cached stat on store::File has no single owner: 4 fields, 5 writers, and Bun.write resets 1 field. That is a PR of its own after this one.
  3. The macOS pipe case has no test.
  4. resolve_size() keeps its name while a file view no longer gets a size from it. A rename is a PR with no behaviour change.

Suites

Debug build, Linux x64, on the pushed source: bun-file-read 41 pass. bun-file-fd-read 6. bun-stdin-slice 10. bun-serve-file 136. body 776. body-clone 86. blob 110. wasm-streaming 33. structured-clone-blob-file 42. expect 416. streams 624. fetch-file-upload 11. bun-file 6. bun-file-exists 1. FormData 149. The Content-Range tests of serve.test.ts 45. bun-write 92 to 96 pass: 1 to 5 tests that spawn a child go over their 5 s limit on this machine, with main's src/ too. fetch.test.ts 374 pass and 22 fail: 21 need IPv6, the public internet or a non-root user, and the 22nd is a memory bound of 256 MB that measures 258 to 265 MB. All 22 fail with main's src/ too. The whole serve.test.ts 329 pass and 2 fail: a root port and /bun:info, which fail with main's src/ too.

Windows x64, debug build, exit codes checked: bun-file-read 27 pass. bun-serve-file 128. body 774. body-clone 84. blob 108. streams 617. bun-file 6. bun-file-exists 1. FormData 149. fetch-file-upload 11. structured-clone-blob-file 38. wasm-streaming 33. expect 416. bun-stdin-slice 5. The Content-Range tests of serve.test.ts 45. Exit code 0 for each file. bun-write 85 pass and 1 fail: Bun.write() without uv_fs_copyfile, which fails on main there too.

bun run rust:check-all passes on 12 targets.


no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/web/fetch/body.test.ts, test/js/web/fetch/body-clone.test.ts, test/js/bun/util/bun-stdin-slice.test.ts, test/js/bun/util/bun-file-read.test.ts, test/js/bun/util/bun-file-fd-read.test.ts, test/js/bun/io/bun-write.test.js, test/js/bun/http/serve.test.ts, test/js/bun/http/bun-serve-file.test.ts

@robobun

robobun commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 7:00 AM PT - Sep 25th, 2026

✅ @robobun, your commit 598cd9d8cdc7132d9bf828721e06fba97e49efb9 passed in Build #120618! 🎉


🧪   To try this PR locally:

bunx bun-pr 43910

That installs a local version of the PR into your bun-43910 executable, so you can run:

bun-43910 --bun

@robobun

robobun commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: draft. The base of this PR is the branch of #43897. Two commits in the diff are copies of #43871. The merge order and the open sibling PRs are in the Notes of the description.

Reproduced on canary 1.4.3+367d939d9 and on a debug build of main (73df7bb), Linux x64:

const f = Bun.file("/proc/self/status");
f.size;                       // or: await f.exists()
await f.text();               // "" (a fresh Bun.file(p).text() returns about 1500 bytes)

const out = Bun.file("out.txt");          // does not exist yet
await out.exists();
await Bun.write(out, Bun.file("in.txt")); // returns 0, out.txt is empty (#4930)

const g = Bun.file("four-bytes.txt");
await g.exists();
await Bun.write(g, "asdfasdf");
await g.text();               // "asdf" (#23902)

All three return the right bytes on this branch. The PR has 67 new tests in bun-file-read.test.ts, bun-write.test.js, bun-file-fd-read.test.ts, bun-stdin-slice.test.ts, serve.test.ts and bun-serve-file.test.ts. A debug build of main fails 42 of them, and the other 25 are guards. This branch passes all 67 on Linux x64, and each one that runs on Windows passes on Windows x64.

This revision is narrower than the first one. slice() on a handle that nothing statted behaves as on main again, and the S3 change moved to #43934.

Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/Blob.rs
Comment thread src/runtime/webcore/Blob.rs
Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/FileReader.rs Outdated
Comment thread src/runtime/webcore/blob/read_file.rs Outdated
Comment thread src/runtime/webcore/blob/read_file.rs Outdated
Comment thread src/runtime/webcore/fetch.rs Outdated
Comment thread src/runtime/server/RequestContext.rs
ReadFile skipped the read when its fstat said st_size 0 and the store's
cached mode said regular file. That cache is only filled once something
stats the shared store, which clone() does since #42053 through
store_reads_repeatably. new Response(Bun.file('/proc/version')).clone()
then made both copies and the Bun.file itself read as "": a procfs file
is a regular file with st_size 0 and real content. Drop the shortcut and
read to EOF. A truly empty file costs one read() that returns 0.
@robobun
robobun force-pushed the robobun/c897cd79/file-stat-not-a-budget branch from 2151da9 to 0af1354 Compare September 25, 2026 10:55
Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/blob/read_file.rs Outdated
@robobun robobun changed the title Bun.file: a cached stat is never the byte budget of a read or a copy Bun.file: do not use a cached stat size as the byte budget of a read or a copy Sep 25, 2026
@robobun
robobun changed the base branch from main to robobun/c897cd79/blob-resolve-size-one-impl September 25, 2026 10:55
…or a copy

`Blob::resolve_size` copied the cached `st_size` of a file onto the blob.
Readers and the file copy of `Bun.write` used it as a byte budget. A
procfs file reports 0 bytes and has content, and a file can grow.

- `resolve_size` stores no size on a file view. A file view keeps
  `MAX_SIZE` ("until EOF") unless `slice()` set a window. `.size`,
  `expect()` and the HEAD `Content-Length` take the length from
  `resolved_size()`.
- After a stat, `ReadFile`, `ReadFileUV` and `FileReader` stop at the
  size from their own `fstat`, so a read after `.size` needs no `read()`
  at EOF. On Windows the stream reader asks `GetFileSizeEx`.
- `slice()` on a handle that nothing statted behaves as before and runs
  no stat. After a stat, an index from the end counts from the length of
  the stat, and `.size` of a window stops at that length.
- `do_sendfile` clamps the window against its own `fstat` before it
  decides if the response is a range.
After .size or exists() on the handle, main sends Content-Length: 200000
and 100000 body bytes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BunFile .text() does not return correct content after BunFile.write() Calling BunFile.exists makes Bun.write write nothing

1 participant