Repository navigation
build: trace exact function entries for the symbol order file, and order on macOS too - #35085
Conversation
…der on macOS too The order file lists ~14k functions that actually run instead of ~38k that share a page with one that does, so the hot set at the front of .text is denser. On a linux-x64 release link: unordered 38,260 KB page-order 31,080 KB (what #33302 ships today) function-order 22,096 KB (-9.0 MB on top of page ordering) The tracer plants INT3 (x86-64) or BRK (arm64) at every nm-listed function start via a writable alias of .text (memfd on Linux, a mach_vm_remap of the COW-promoted __TEXT on macOS), restores it on first SIGTRAP, and records the address. pthread_sigmask/sigprocmask are interposed to keep SIGTRAP unblocked on background threads (mimalloc's scavenger blocks every signal, and a blocked synchronous SIGTRAP makes the kernel reset the handler to SIG_DFL). macOS arm64 release now links with Apple ld's -order_file, driven by the same two-pass trace-and-relink the Linux lanes already use. The CI plumbing (inheritOrderFile / regenerateOrderFile / verifyOrderFileApplied) is unchanged; usesOrderFile now covers darwin, linkDepends lists the file there, and the seed comment no longer says lld-only. [generate symbol order]
WalkthroughAdds macOS arm64 support to the linker symbol-order-file optimization, replaces the Linux-only page-fault tracer with a cross-platform breakpoint-based function-entry tracer (functrace.c), rewrites the order-file generator pipeline accordingly, updates CI artifact inheritance/upload logic, and adds a Buildkite trace-order pipeline step. ChangesCross-platform symbol order file feature
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
|
Updated 10:10 AM PT - Jul 22nd, 2026
❌ @robobun, your commit 3324259 has 2 failures in
🧪 To try this PR locally: bunx bun-pr 35085That installs a local version of the PR into your bun-35085 --bun |
ld64.lld supports -order_file, so a cross-compiled darwin arm64 build can link with an inherited one even though it cannot trace its own binary. usesOrderFile is about consuming the file, not producing it; canTraceOrderFile already gates generation on canRunOnHost.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
- Open the trace record before planting breakpoints, so an open/mmap failure leaves no breakpoints live (the handler dereferenced a NULL record). - On macOS, flip __TEXT back to RX before remap_executable() returns either way; the RW alias stays writable, so install_breakpoints() and the handler write through it. Nothing fallible sits between RW and RX anymore. - On macOS, the sigaction interposer now calls sigaction() directly instead of a real_sigaction pointer that was NULL until late in init: dyld applies the __interpose table before any constructor runs, and it does not redirect the interposing dylib's own calls. Verified on both platforms that a missing starts file or an unwritable output path lets the workload run to completion untraced.
The functrace.c and ptyrun.c macOS paths (mach_vm_protect/remap, __DATA,__interpose, util.h forkpty, DYLD_INSERT_LIBRARIES) were only verified by hand. Gate both describe blocks on the same linux || darwin-arm64 predicate the generator uses and branch the compile flags and preload variable the way generate.ts does.
The generator never runs on musl (bun-musl links statically, so LD_PRELOAD cannot load the tracer; usesOrderFile already excludes it), so compiling and running the tracer there exercises nothing the build uses. The describe gate now matches that: linux glibc or darwin arm64.
INT3 is a trap, so on x86-64 RIP is already past it and returning from the handler would resume there rather than re-execute under the new default disposition. raise() re-delivers to the calling thread regardless of trap-vs-fault semantics.
…cross-build The darwin-aarch64 build lane cross-compiles from debian, so it cannot run the binary it links and cannot trace an order file. A new darwin-aarch64-trace-order step runs on the bare-metal mac queue after build-bun, downloads bun-profile, runs scripts/orderfile/generate.ts against it, and uploads the .order artifact. One build of lag: build N's trace is consumed by build N+1's link. Non-PR only (orderFileEligible ignores PRs) and soft-fail (an order file is an optimization). inheritOrderFile now searches every step of a previous build rather than only the same step key, since the artifact for a cross-compiled lane comes from the sibling trace step. The artifact name is target-unique so this is unambiguous. reportOrderFileCannotTrace's message now says it is expected once while the trace step seeds the chain.
…bol order] Always on main, and anywhere else when the commit subject carries [generate symbol order] (the same tag ci.ts already honours for forcing a canary to trace its own order file). Lets a PR that changes the tracer prove the step works before merge.
…on error [generate symbol order] - packageAndUpload() only re-uploads the standalone .order when this lane traced it itself (canTraceOrderFile). A cross-compiled lane's fresh trace comes from the sibling trace-order step; re-uploading the inherited copy there would give inheritOrderFile() two same-named artifacts whose download order is undefined. - OrderFileContext.stepKey is no longer read (inheritOrderFile searches every step of a build now); drop the field, its population, and the test fixture. - functrace.c's open_record/read_starts/remap_executable close their fd when the step after open() fails, matching what the deleted pagetrace.c did.
Every build lane runs on the aarch64 buildHostPlatform (getLinkBunAgent), so linux-x64's canRunOnHost is false and it cannot trace itself either. Without a trace step its .order chain has zero publishers once pre-PR artifacts age out. traceOrderTargets lists each cross-compiled order-file target alongside the test platform to trace it on; getTraceOrderStep takes that platform and uses getTestAgent for the agent (bare-metal queue on darwin, EC2 on linux). The trace host's image dep goes on the step itself, same as verify-baseline. linux-aarch64 stays absent: its build lane is native and packageAndUpload is its sole publisher.
The predicate matched the linux-x64 asan build platform too (no abi, same os/arch), emitting a step that tries to download bun-linux-x64-profile.zip from the asan lane, which uploads bun-linux-x64-asan.zip instead. usesOrderFile is false under a sanitizer anyway.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/build/ci.ts`:
- Around line 429-433: Condense the comments at scripts/build/ci.ts lines
429-433, 715-718, and 849-853 to three lines or fewer each, preserving their
essential rationale about artifact publication, trace seeding, and no---step
downloads; make no code changes.
In `@scripts/orderfile/generate.ts`:
- Around line 136-139: Update readSymbolTable so macOS does not invoke
/usr/bin/nm with unsupported GNU-only flags: prefer a PATH- or xcrun-resolved
llvm-nm when available, or remove --defined-only and --numeric-sort for the
cctools fallback while preserving equivalent filtering and numeric sorting in
process.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 48ea718f-3682-4ea8-aac3-43d1b12a5f0b
📒 Files selected for processing (11)
.buildkite/ci.mjsscripts/build/ci.tsscripts/build/configure.tsscripts/build/flags.tsscripts/orderfile/functrace.cscripts/orderfile/generate.tsscripts/orderfile/pagetrace.cscripts/orderfile/ptyrun.ctest/js/bun/perf/functrace-fixture.ctest/js/bun/perf/linker-order.test.tstest/js/bun/perf/pagetrace-fixture.c
💤 Files with no reviewable changes (2)
- test/js/bun/perf/pagetrace-fixture.c
- scripts/orderfile/pagetrace.c
…bol order] readSymbolTable's regex already filters to defined text symbols and nothing depends on nm's output order, so --defined-only and --numeric-sort were redundant. Dropping them removes the one place a cctools nm would have differed from the llvm-nm modern macOS ships as /usr/bin/nm.
…st [generate symbol order] read_starts() routes header validation and the calloc through the same branch so a single munmap handles both error paths, matching the fd-leak treatment in be15cb1. The test's sibling nm drops --defined-only to match readSymbolTable() and checks nm's exit so a failure names itself.
There was a problem hiding this comment.
No new findings after 3e2d626, but this replaces the page-fault tracer with a ~540-line breakpoint-injection preload (functrace.c: signal interposition, memfd/mach_vm_remap text aliasing, self-modifying code across three arch/OS combos) and rewires the cross-build .order artifact chain — worth a human look.
What was reviewed:
functrace.cerror paths — fd/mmap release now paired on every early return; macOSregion_countnever exceedsMAX_REGIONS(single__textsection).- CI publisher topology — one
.orderpublisher per target after e34d55f/56d6cee (linux-aarch64 self-traces; linux-x64 + darwin-aarch64 via-trace-order; asan/musl excluded). nminvocations — GNU-only flags dropped in bothreadSymbolTable()and the test's sibling call; exit code now asserted.usesOrderFile/linkDepends/linker-flag gating — darwin and linux flags mutually exclusive, order file listed as a link dep on both.
Extended reasoning...
Overview
Replaces pagetrace.c (page-fault tracer) with functrace.c (function-entry tracer via INT3/BRK breakpoints), extends symbol ordering to macOS arm64 via -order_file, and adds -trace-order CI steps so cross-compiled targets (darwin-aarch64, linux-x64) can trace on their native test fleet. Touches .buildkite/ci.mjs, scripts/build/{ci,configure,flags}.ts, scripts/orderfile/{functrace.c,generate.ts,ptyrun.c}, and the test file.
Security risks
None user-facing. functrace.c is a build-time preload that runs only during CI's order-file generation against a binary CI itself just built; it never ships in bun. It does remap the executable's text segment RW and plant breakpoints, but only inside a controlled CI step with soft-fail semantics.
Level of scrutiny
High. The new C file is genuinely intricate systems code — sigaction/sigmask interposition to keep SIGTRAP deliverable on background threads, dual-mapped text via memfd (Linux) and VM_PROT_COPY + mach_vm_remap (macOS), atomic instruction restore + icache sync in a signal handler, PC rewind across three arch/OS combos. The CI artifact-inheritance chain (which lane publishes, which inherits, one-build lag) also had several real bugs during review (missing linux-x64 publisher, asan-profile match, double-publisher race) that were caught and fixed across four iterations. That history argues for a maintainer sign-off on the final shape rather than auto-approval.
Other factors
All prior inline findings are resolved and verified in the current diff. Test coverage is reasonable for the wiring (usesOrderFile, linkDepends, flag emission for both platforms, tracer round-trip on a fixture with an exec'd child). The functional claim (−9 MB RSS on linux-x64, first macOS ordering) is a build-quality optimization whose failure mode is a correct-but-unordered binary plus a Buildkite warning annotation, so downside risk is bounded — but the mechanism is novel enough that someone familiar with the mach_vm/memfd paths should confirm the remap approach.
The macOS pty line discipline echoes end-of-input as the two printable characters ^D followed by two backspaces (ECHOCTL), which landed on the front of the probe's line. Collapsing every control-character run to a newline separates it, and does the job of the old \r strip too.
There was a problem hiding this comment.
No issues found this run; all earlier review threads are resolved. Deferring to a human because the change is large and intricate: a new ~540-line breakpoint tracer that remaps the executable's text and interposes sigaction/sigmask on two platforms, plus CI artifact-publication topology that decides which release lanes get an order file.
What was reviewed:
functrace.cresource lifetimes on every error path (fd/memfd/mmap now paired), the not-ours SIGTRAP re-delivery, and theread_startsbounds/duplicate/already-BRK filtering.- CI publisher topology: exactly one
.orderpublisher per target (linux-aarch64 self-traces; linux-x64 and darwin-aarch64 via-trace-order), release-profile-only predicate, andinheritOrderFile()no---stepdownload. usesOrderFile/linkDepends/linker-flag gating — each platform gets only its own flag, and neither on debug/sanitizer links.- The test's nm invocation now matches
readSymbolTable()(barenm, exit checked) and the pty ^D echo is stripped on macOS.
Extended reasoning...
Overview
Replaces the page-fault order-file tracer with a function-entry tracer (scripts/orderfile/functrace.c, new, ~540 lines) and extends symbol ordering to macOS arm64. Touches: the tracer + generator + ptyrun (scripts/orderfile/), build-flag wiring (flags.ts, configure.ts), CI artifact flow (ci.ts, .buildkite/ci.mjs — new -trace-order steps and publisher gating), and the test file + fixtures. Deletes pagetrace.c and its fixture.
Security risks
None that reach the shipped binary. functrace.c is a build-time preload that runs only during CI's trace step against bun-profile; it does self-modifying-code and signal interposition, but on a process the build owns. The generator reads nm output and a fixed starts file it wrote itself; no untrusted input crosses into the runtime.
Level of scrutiny
High. Nothing here executes in the shipped bun, but the CI topology it introduces decides whether release binaries link ordered at all — an earlier iteration silently dropped the linux-x64 publisher, which would have cost the −16 MB win once older artifacts aged out. The tracer itself is platform-specific low-level code (memfd/MAP_FIXED remap on Linux, mach_vm_protect COPY + mach_vm_remap on macOS, sigaction/pthread_sigmask interposition) whose failure mode is a soft-failed step rather than a broken build, but correctness of the Mach-O __TEXT,__text-only bounding and the RW-alias write path deserves a human read.
Other factors
The PR has been through four review rounds; every finding I raised (fd/mmap leaks, linux-x64 publisher gap, asan-profile predicate, sibling nm --defined-only) is now fixed and the threads resolved. Test coverage is reasonable for the config-level wiring and the tracer's exec-survival property, and verifyOrderFileApplied() gives a runtime check on release links. Still, the combination of a new native tracer, two-platform VM tricks, and CI inheritance-chain changes is well past the "simple/mechanical" bar for auto-approval.
|
Builds 77715 and 77858 (retrigger): the diff is green.
Remaining red is unrelated to this change: a |
There was a problem hiding this comment.
No new findings this pass — the earlier rounds' issues (linux-x64 order-file publisher, asan trace-order predicate, fd/mmap error-path leaks, sibling nm invocation) are all addressed and CI is green. That said, this is a ~540-line platform-specific C tracer (memfd/mach_vm remapping, sigaction/sigmask interposition, breakpoint injection) plus new CI artifact-inheritance topology across three targets, so it's worth a human look before merge.
What was reviewed:
functrace.csignal handler / remap paths on both linux and darwin arms; error-path resource release now matches acquisition- CI wiring: single-publisher invariant for each
.orderartifact (linux-aarch64 self-traces, linux-x64/darwin-aarch64 via-trace-order), release-profile-only predicate,inheritOrderFileno---stepsearch usesOrderFile/linkDepends/linker-flag gating for darwin arm64 vs linux — neither flag leaks to the other platform or to debug/sanitizer linksgenerate.tsnm invocation and the test's sibling nm call now both use barenmwith regex filtering
Extended reasoning...
Overview
Replaces the page-fault order-file tracer with a function-entry tracer and extends symbol ordering to macOS arm64. Touches: a new ~540-line scripts/orderfile/functrace.c (INT3/BRK breakpoint injection, memfd/mach_vm_remap RW-alias of .text, sigaction/pthread_sigmask/sigprocmask interposition), a rewritten scripts/orderfile/generate.ts, ptyrun.c macOS support, usesOrderFile()/linkDepends()/linker flags in scripts/build/flags.ts, artifact publication/inheritance in scripts/build/ci.ts, new -trace-order steps in .buildkite/ci.mjs, and the test file + fixtures.
Security risks
None user-facing. functrace.c is a build-time preload that never ships in bun; it only runs against bun-profile on CI/local release relinks. It does remap the traced executable's text and interpose signal APIs, but only within the CI trace step's own process. The starts/trace file formats are magic+version-checked and count-bounded before use.
Level of scrutiny
High. This is not runtime code, but it directly shapes the shipped release binary's .text layout on three targets and adds a cross-build artifact-inheritance chain (one-build lag, single-publisher-per-target) that, if it silently breaks, costs ~16 MB RSS with no build failure. The C tracer is intricate platform-specific code (Mach VM COW promotion, dyld interposition, arm64 icache sync, signal-mask stripping for mimalloc's scavenger) that a maintainer should read once end-to-end.
Other factors
This PR has been through ~8 review iterations on this thread; every finding I raised (unknown-trap re-delivery, darwin trace-order seeding, linux-x64 publisher gap, asan-profile predicate, fd/mmap error-path leaks, sibling --defined-only flag) was fixed and is now resolved. The bug-hunting system found nothing on the current head. robobun reports build 77715 green including the new linker-order.test.ts on darwin-aarch64 and both new -trace-order steps uploading real artifacts. Test coverage is reasonable (flag gating, linkDepends, tracer end-to-end on a fixture, ptyrun on both platforms). Deferring because the scope — new low-level tracer + CI topology change — is beyond what should land without a human sign-off, not because of any open concern.
…sed-ci-images The one conflict was .buildkite/ci.mjs: main added the symbol order-file trace step (#35085) to the pipeline generator, which this branch had reduced to a two-line shim that hands off to .buildkite/ci.ts. Keep the shim and port main's two hunks — the traceOrderTargets list with getTraceOrderStep(), and the trace-order call site in getPipeline() — verbatim into ci.ts, where the generator now lives. Every helper the ported code calls already exists there.
2 participants
What
Replaces the page-fault order-file tracer with a function-entry tracer (INT3 on x86-64, BRK on arm64) and extends symbol ordering to macOS arm64 via
-order_file.The page tracer lists every function that shares a 4 KB page with one that runs, so the order file carried ~38k names of which ~23k never execute; those cold functions sat at the front of
.textand diluted the hot set. Recording exact entries lists only the ~14k that actually run.Result
Linux x86-64 release,
bun -eRSS (median of 5):So about 9 MB on top of what #33302 already ships, for the same-size binary.
macOS arm64 had no ordering at all before this. The tracer was verified against a release
bun-profileon a darwin-arm64 box: 9,612 functions traced across the eight workloads, first entries_mi_process_attach → _mi_lock_acquire → ...as expected.How
scripts/orderfile/functrace.cis a preloaded library (LD_PRELOAD/DYLD_INSERT_LIBRARIES) that:PT_LOAD Xsegment on Linux, the__TEXT,__textsection on macOS, so the Mach-O header and read-only datanmlists asTare never touched),VM_PROT_COPY+mach_vm_remapon macOS, since RWX on__TEXTis refused even with COPY),nm, drops anything outside the text bounds or already a breakpoint (JSC's LLInt places literalint3at never-taken bytecode labels), plants a breakpoint at each,sigactionis interposed so nothing replaces the SIGTRAP handler.pthread_sigmask/sigprocmaskare interposed to strip SIGTRAP from any block set: mimalloc's scavenger thread blocks every signal, and a blocked synchronous SIGTRAP makes the kernel reset the handler toSIG_DFLand kill the process, which would end the trace at the first breakpoint that thread touches.scripts/orderfile/generate.tskeeps the same eight workloads and the samegenerateOrderFile()/runCommand()exports. It now writes a starts file fromnm --defined-only, runs each workload underfunctrace.c, and maps recorded addresses straight back to names; no page-to-function walk.Wiring
usesOrderFile()now returns true for darwin arm64 too (both Apple ld and ld64.lld take-order_file).-Wl,-order_file,<buildDir>/linker.orderflag entry sits next to the existing--symbol-ordering-fileone, gated to darwin.linkDepends()lists the order file on darwin so regenerating it relinks, and only relinks.ptyrun.csetsDYLD_INSERT_LIBRARIESon macOS so the tty workload still reaches the tracer.Every build lane cross-compiles from the aarch64
buildHostPlatform, so only linux-aarch64 can trace its own binary. For the other two order-file targets a new-trace-orderstep (.buildkite/ci.mjs) runs on the target-arch test fleet afterbuild-bun, downloadsbun-profile, runsgenerate.tsagainst it, and uploads the.orderartifact:darwin-aarch64-trace-orderon the bare-metal mac queuelinux-x64-trace-orderon the x64 debian test agentinheritOrderFile()now searches every step of a previous build (the artifact name is target-unique), so build N+1's cross-compile link picks up build N's trace.packageAndUpload()only re-uploads the standalone.orderfrom lanes that traced it themselves, so each target has exactly one publisher. Non-PR only and soft-fail, since the order file is an optimization.Tests
test/js/bun/perf/linker-order.test.tscovers:usesOrderFile/linkDepends/ the linker flag for both linux and darwin arm64, and that neither flag appears on the other platform or on debug/sanitizer linksfunctrace.centry count surviving a child exec, on linux glibc and darwin arm64 (replaces the old page-tracer fixture)no test proof · iteration 5 · Platform-specific test-only change; deferring to CI.