Conversation
|
Status: draft, stacked on #37082. This revision (1924ab8) sends such a file through the reader that How to reproduce on main or bun 1.4.2, Linux: await using server = Bun.serve({ port: 0, fetch: () => new Response(Bun.file("/proc/version")) });
const res = await fetch(server.url);
console.log(res.headers.get("content-length"), (await res.text()).length);
// main: "0" 0
// this PR: "219" 219The same happens for a static route. #44064 tracks the parts that this PR does not change. |
5561b9f to
86b62a0
Compare
|
Updated 6:46 PM PT - Sep 28th, 2026
❌ @robobun, your commit 1924ab8 has 1 failures in
🧪 To try this PR locally: bunx bun-pr 43974That installs a local version of the PR into your bun-43974 --bun |
86b62a0 to
64489d0
Compare
A regular file on procfs or cgroupfs reports st_size 0 and has content. do_sendfile and FileRoute framed the response from st_size, so they sent Content-Length: 0 and no body. Such a file now takes the arm of a pipe: FileResponseStream reads it to EOF, with no Content-Length, Range or validator from the stat. A slice passes its window as the limit of the reader. HEAD, TRACE and a descriptor of the caller keep the answer of the stat.
64489d0 to
1924ab8
Compare
|
@alii this PR needs one decision from you, and it is the decision that is open on #41593 too. A regular file on procfs or cgroupfs reports
Shape A is on this branch. Shape B is on Questions:
|
Part of #44064
Stacked on #37082. Questions for alii or Jarred-Sumner:
Blobsize)?Problem
Bun.serveanswersContent-Length: 0and no body fornew Response(Bun.file("/proc/version")). The file has 219 bytes.do_sendfile(RequestContext.rs:1857) andFileRoute::serve(FileRoute.rs:335) frame fromst_size. procfs and cgroupfs report 0.Fix
st_size0 takes the arm of a pipe:FileResponseStreamreads it to EOF, and the producer writes noContent-Length, Range or validator. Linux only.BodyLength::UpTo, the limit of the reader.bun-serve-file.test.ts(26 new tests, 21 fail on the base). Alsoserve-directory-routes,bun-serve-static,serve-http2.Background
FileResponseStreamowns each file body ofBun.serve. A body with no length ends at EOF: chunked over HTTP/1.1, with the last chunk that Bun.serve: terminate FIFO/pipe file responses at EOF #37082 adds.Content-Length,error()on a read error): a second reader, 428 lines, a 256 KiB cap.Downsides
BunFileafter.sizeorexists()(Bun.file: do not use a cached stat size as the byte budget of a read or a copy #43910),Bun.file(fd),{ dir }routes.read, 1 to 2 allocations, and itsLast-Modifiedon a static route..textgrows by 1,536 bytes./now sends/proc/self/environ. The read holds the JS thread: 179 ms for/proc/pagetypeinfo.Notes
What the server sends
Raw HTTP/1.1,
Connection: close, Linux x64, release builds. Base: #37082 at 26376b9. Revision before: 64489d0, which read the file before the status line./proc/versionhas 219 bytes,/proc/cpuinfo177,934 and/proc/kallsyms12.5 MB here./proc/version, fetch handler and static routeContent-Length: 0Content-Length: 219Content-Length: 219Bun.file("/proc/version").slice(10, 60)Content-Length: 0Content-Length: 50Content-Length: 50/proc/cpuinfo(65 reads)Content-Length: 0Content-Length: 177934/proc/kallsymsContent-Length: 0Content-Length: 0Range: bytes=0-9416,Content-Range: bytes */0200, whole body200, whole bodyContent-Length: 0Content-Length: 0Content-Length: 0Content-Length: 0,Last-ModifiedContent-Length: 0,Last-ModifiedContent-Length: 0/proc/self/mem(the first read fails with EIO)200500througherror(), or the next handler/proc/versionContent-Length: 0Content-LengthContent-Length: 0Content-Length: 0Content-Length: 0Content-Length: 0BunFileafter.sizeorawait exists()Bun.file(fd), two timesIf-None-Match: *, static route304,Last-Modified304304A read error gets the answer that a file with a size gets on the base: the connection closes.
Measurements
Release builds of the base (26376b9) and of this PR (1924ab8), same machine and toolchain.
strace,perfandvalgrindare not installed on the test machine andperf_event_openis not permitted there. The counts come from aptracesyscall counter, aptracesingle-step counter andgdbbreakpoints on the mimalloc entry points.User instructions per request from the return of
recvfromto the first send (single-step count, 20 requests after 40 warm-up requests):/proc/versionGET, static route/proc/versionGET, fetch handler (median)The route counts are equal in all 20 requests. The fetch handler runs JavaScript, so its count changes with the work of the garbage collector. By function, for a sized file:
FileRoute::on+11,do_sendfile+4,FileResponseStream::start+5.Syscalls per request on one keep-alive connection, (count at 300 requests minus count at 100) / 200,
futexleft out:setImmediatesendto)sendto)sendfile): fetch handler, static routereadthat returns 0)/proc/versionGET: fetch handler, static routeread)/proc/versionGET, handler that resolves aftersetImmediatesendto)read, 9sendto)/proc/versionHEAD: fetch handler / static route/proc/cpuinfoGET, fetch handlerread, 8sendto)Heap allocations per request (
gdbbreakpoints onmi_malloc,mi_calloc,mi_zalloc,mi_reallocand the aligned and heap variants, count at 500 requests minus count at 100, divided by 400). Two runs of the same build differ by up to 0.2 on a route and 0.5 on a handler:/proc/versionGET, static route/proc/versionGET, fetch handlerA file with no size allocates what a file with a size allocates: the
FileResponseStreamand the task that closes the descriptor.Time that one GET holds the event loop (greatest gap of a 1 ms timer, median of 15 requests, uid 0, load average over 400):
/proc/version(219 B)/proc/self/smaps_rollup(698 B, mode 0444) at 0.03, 0.5, 1 and 2 GiB resident/proc/cpuinfo(178 KB)/proc/slabinfo(45 KB, mode 0400)/proc/kallsyms(12.5 MB)/proc/pagetypeinfo(3.9 KB, mode 0400)/proc/vmallocinfo(2.2 MB, mode 0400)The time is the time the kernel takes to make the content. The reader of
FileResponseStreamrunsreadon the JS thread for each file that does not go throughsendfile, on the base too. The four Dockerfiles indockerhub/set noUSER, so the images run as uid 0, which can read the files of mode 0400.Binary size:
.textofbun-profile(size)bun, stripped, file bytesFrom
nm -S:do_sendfile+184 in each of its 8 copies,FileRoute::on+98,FileResponseStream::start+29.StartOptionsandFileResponseStreamkeep their size:BodyLengthhas the 16 bytes thatOption<u64>had.Source: +91 -21 lines in 4 files.
Tests
test.todointest/js/bun/http/bun-serve-file.test.ts("Bun.file() of a regular file whose stat size is 0"), Linux only. On a release build of the base, 21 fail. The 5 others are guards, with a comment that says so: an empty file (4) andBun.file(fd)(1)./proc/versionand two files of a child process that stopped itself with SIGSTOP. Itsenvironhas about 480 KB. Itsmapshas 64 added mappings and comes in reads of at most one page.Content-Lengthor when chunks have no last chunk.Last-Modified.bun-serve-file.test.ts,serve-directory-routes.test.ts,serve-file-slice-read-error.test.ts,bun-serve-static.test.ts,serve-http2.test.ts,serve-if-none-match.test.ts.cargo clippy -p bun_runtimeandbun run rust:check-allpass.bun-serve-file.test.tsthat this PR does not change ("pollable Bun.file ...") start a debug build of bun and ended at the 5 s limit there, with the source of main too.Limits
stat:Content-Length: 0. Bun.serve: resolve a file body for HEAD the same way as for GET #41585 owns HEAD.BunFilewhose size something asked for holds a window of 0 bytes:.size,await exists(), a read ofresponse.body,Bun.inspect(response). The server cannot tell it fromslice(0, 0).stat(),lastModifiedandclone()do not store it. Bun.file: do not use a cached stat size as the byte budget of a read or a copy #43910 and Leave a body stream unsized when the file's st_size is 0 #41593 own that.Bun.file(fd)keeps the answer of itsstat. A read moves the position of a descriptor that the caller owns.{ dir }route keeps the answer of itsstat. There the client names the file. A maintainer decides if such a route reads these files./proc/kmsg,trace_pipe) returns EAGAIN. No such file is readable on the test machine, so this is from the code and not from a run: the reader does not wait for a regular file (src/io/PipeReader.rs, "Received EAGAIN while reading from a file"), so the response stays open until the idle timeout. The base answers an empty200at once.Last-Modifiedand the304forIf-Modified-Since. The server cannot tell it from a file of procfs before it reads.slice()counts from the largest size of aBlob, as inBun.file(path).slice(0, -10).bytes(). Bun.file().slice(): count a negative index back from the file's real end #41257 owns that.st_size4096 and holds fewer bytes. It is not in this class. Close the connection when a Bun.file response ends short of its Content-Length #41622 covers a body that ends short of itsContent-Length.respondWithFileofnode:http2frames fromstat.sizetoo. node:http2: handle an empty byte range in respondWithFile and respondWithFD #41516 covers it.reports_no_sizeis false there.Self-review
The review of the revision before raised 45 concerns. Its two blockers asked for this shape: one reader, two producers, after #37082. The directory route, the answer for HEAD and the read before the status line left the PR. Not taken:
bun_systhat the sibling PRs can call.reports_no_sizestays in the server. The reads of the siblings are in other crates and other PRs.FileResponseStream, and c600196 fixed the same class inreadFileSync.serve-file-slice-read-error.test.tscovers them for a file with a size.Other PRs
FileResponseStream::finishends withresp.end(b""), which writes the last chunk.do_sendfile. It changes the same lines ofdo_sendfile.Blob::resolve_size. #43910 changesRequestContext.rsand this test file.Bun.filereads,fetchbodies, the module loader, dotenv, FormData. No file in common.Bun.file().stream()on the work pool. It does not changeFileResponseStream.Found on the way, not in this PR: a comment in
uws_res_end_without_body(src/uws_sys/libuwsockets.cpp) namesFileResponseStream::finishas a caller. With #37082 it is not one.The branches
robobun/00c06367/serve-file-whole-body(64489d0) androbobun/00c06367/serve-file-chunked-body(86b62a0) hold the two shapes that read the file before the status line.