Skip to content

Report OutOfMemory crashes from infallible allocation failure - #31690

Closed
robobun wants to merge 4 commits into
mainfrom
farm/afc126d6/oom-alloc-error-hook
Closed

robobun wants to merge 4 commits into
mainfrom
farm/afc126d6/oom-alloc-error-hook

Conversation

@robobun

@robobun robobun commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator

Problem

Out-of-memory crashes from ordinary (infallible) allocations are invisible: zero OutOfMemory-typed crash reports come from Rust builds, while Zig stable builds report thousands. In Zig, every failed allocation returned error.OutOfMemory and bubbled to bun.outOfMemory() → CrashReason::OutOfMemory → auto-report. In the Rust port, only explicit Result<_, AllocError> paths go through bun_core::handle_oom; plain Vec/Box/String growth failure goes to the global allocator's handle_alloc_error, whose std default hook prints:

memory allocation of 2305843009213693952 bytes failed

and calls abort() — SIGABRT (exit 134), no "Bun ran out of memory" message, no trace string, no report URL, no upload. Since the vast majority of allocations are the infallible kind, effectively all real-world OOMs vanish from telemetry. The POSIX crash handler only registers SIGSEGV/SIGILL/SIGBUS/SIGFPE, so nothing downstream catches it either (and by SIGABRT time the OOM context would be gone anyway).

Fix

install_hooks() — where std::panic::set_hook is already wired — now also registers std::alloc::set_alloc_error_hook (nightly API; the workspace is pinned to nightly). The hook routes into the existing out_of_memory() → crash_handler(CrashReason::OutOfMemory, …) path, so infallible allocation failure produces the same report as the explicit AllocError path and Zig:

oh no: Bun has run out of memory.

To send a redacted crash report to Bun's team,
please file a GitHub issue using the link below:

 https://bun.report/1.4.0/…

The hook runs under memory pressure, so it must not allocate: its body is a single diverging call, and the crash handler itself writes through stack buffers and raw stderr. A second OOM inside the report path hits the existing PANIC_STAGE re-entry guard and aborts instead of looping. This also covers the in-tree direct handle_alloc_error callers (VirtualMachine.rs, analyze_transpiled_module.rs).

(-Zoom=panic was considered and rejected: it would type these as Panic instead of OutOfMemory; registering SIGABRT was rejected because the OOM context is gone by signal time.)

Test

New crash_handler.infallibleOutOfMemory binding in bun:internal-for-testing performs a real Vec::with_capacity(1 << 61) — a reservation no 64-bit address space can satisfy — so the failure travels the genuine route: global allocator returns null → RawVec → handle_alloc_error → hook → report. Wired into the existing automatic crash reporter matrix in test/cli/run/run-crash-handler.test.ts, which asserts the OOM message, the report URL, and the POST arriving at a self-hosted report server. Under ASAN the child gets ASAN_OPTIONS=allocator_may_return_null=1 so the interceptor returns null instead of hard-erroring before Rust sees the failure.

Verified on the ASAN debug build: test fails without the src changes (fixture reports the approach as unknown, process exits 0 with no report) and passes with them; the other crash-reporter approaches (panic, segfault, outOfMemory) are unaffected.

Cargo.lock: syncs the bstr dependency bun_bin gained in #31668 (lockfile wasn't regenerated there; any cargo invocation reproduces this hunk).

@coderabbitai

coderabbitai Bot commented Jun 2, 2026 •

Copy link
Copy Markdown
Contributor

PR changed again? Review this PR in Change Stack to compare snapshots and stay oriented.

Review Change Stack

Warning

Review limit reached

@robobun, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 51 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6856b68b-9f91-4ff6-a67e-3775c2e97a07

📥 Commits

Reviewing files that changed from the base of the PR and between 3a4863d and aa66139.

📒 Files selected for processing (1)
  • test/cli/run/fixture-crash.js

Walkthrough

This PR adds a Rust allocation error hook that catches infallible allocation failures and routes them through Bun's crash-handler system. It enables the nightly alloc_error_hook feature, implements a hook function that delegates to the existing out_of_memory() path, registers the hook during initialization, adds a JavaScript testing API, and extends tests to verify consistent crash reporting across both OOM pathways.

Changes

Allocation Error Hook and Testing

Layer / File(s) Summary
Allocation error hook implementation and registration
src/crash_handler/lib.rs
Enable nightly alloc_error_hook feature gate, implement a cold allocation error hook function that delegates all failures to out_of_memory(), and register the hook via set_alloc_error_hook during crash-handler initialization.
Allocation error path documentation
src/crash_handler/handle_oom.rs
Update doc comment to clarify that global allocator OOM handling is now intercepted by the alloc-error hook and routed through out_of_memory(), and document explicit Result<T, AllocError> threading cases.
JavaScript testing API for allocation errors
src/js/internal-for-testing.ts, src/runtime/api/crash_handler_jsc.rs
Add infallibleOutOfMemory() method to exported crash_handler testing bindings type, extend JSC host-function dispatch table with dynamic entry count, and implement js_infallible_out_of_memory host function that suppresses core dumps and triggers a deliberately large allocation to exercise the allocator error path.
Test coverage for allocation error hook
test/cli/run/fixture-crash.js, test/cli/run/run-crash-handler.test.ts
Extend crash-handler tests to include infallibleOutOfMemory approach, conditionally inject ASAN allocator options, generalize assertions so both outOfMemory and infallibleOutOfMemory produce identical "out of memory" crash signatures, and update fixture documentation.
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely summarizes the main change: adding OutOfMemory crash reporting for infallible allocation failures, which is the primary objective of this PR.
Description check ✅ Passed The description provides comprehensive information covering problem statement, solution approach, testing methodology, and verification details, exceeding the template requirements.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions Bot added the claude label Jun 2, 2026
@robobun

robobun commented Jun 2, 2026 •

Copy link
Copy Markdown
Collaborator Author

Comment thread src/runtime/api/crash_handler_jsc.rs

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the capacity-hint nit. The change looks correct and well-tested, but since it installs a process-wide alloc_error_hook (nightly API) that alters how every infallible allocation failure terminates, I'd like a human to sign off on the approach.

Extended reasoning...

Overview

This PR registers std::alloc::set_alloc_error_hook in crash_handler::install_hooks() so that ordinary Vec/Box/String allocation failures route into the existing CrashReason::OutOfMemory crash-report path instead of std's default "memory allocation of N bytes failed" + abort(). It adds the #![feature(alloc_error_hook)] nightly gate, a 4-line hook that calls out_of_memory(), a doc-comment update in handle_oom.rs, a new test-only infallibleOutOfMemory binding in crash_handler_jsc.rs (plus the ENTRIES.len() fix from my earlier nit), and wires it into the existing run-crash-handler.test.ts matrix with appropriate ASAN_OPTIONS handling. The Cargo.lock hunk is an unrelated lockfile resync.

Security risks

None apparent. No new attack surface — the hook only fires on allocation failure and routes into pre-existing crash-report machinery. The new JS binding is gated behind bun:internal-for-testing (debug/CI only). Crash report uploads already existed for the explicit OOM path; this just makes the infallible path produce them too.

Level of scrutiny

Moderate-to-high. While the diff is small and the implementation looks correct, it changes process-global behavior: every infallible allocation failure across the entire binary now flows through out_of_memory() → crash_handler() instead of std's default abort. The hook runs under genuine memory pressure, so the no-allocation invariant of the downstream crash path matters (the description argues it holds via stack buffers + raw stderr + the PANIC_STAGE re-entry guard, which seems sound). It also adds a new nightly feature dependency. The PR description explicitly weighs and rejects alternatives (-Zoom=panic, SIGABRT handler), which is exactly the kind of design decision a human maintainer should ratify.

Other factors

The earlier nit (capacity hint vs. ENTRIES.len()) was addressed in 87f67ac. The bug-hunting pass found nothing on the latest revision. Test coverage is solid: the new case exercises the real allocator → handle_alloc_error → hook chain end-to-end and asserts the OOM message, report URL, and POST receipt. Given the global scope of the behavior change and the nightly-feature addition, I'm deferring rather than auto-approving.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the capacity-hint nit. I didn't find any further issues, but since this registers a process-global alloc_error_hook (new nightly feature gate) that changes how every infallible allocation failure terminates, it's worth a human sign-off on the approach.

Extended reasoning...

Overview

This PR routes infallible allocation failures (Vec/Box/String growth → handle_alloc_error) into Bun's crash reporter by registering std::alloc::set_alloc_error_hook alongside the existing panic hook in install_hooks(). It adds the #![feature(alloc_error_hook)] nightly gate to src/crash_handler/lib.rs, a new alloc_error_hook fn that diverges into out_of_memory(), an updated doc comment in handle_oom.rs, a new infallibleOutOfMemory test-only JS binding (crash_handler_jsc.rs / internal-for-testing.ts), and a new case in the existing crash-reporter test matrix. The Cargo.lock hunk is an unrelated lockfile sync for bstr. My earlier nit (the create_empty_object capacity hint drifting to 8 vs 9 entries) was addressed in 87f67ac by hoisting ENTRIES and passing ENTRIES.len().

Security risks

None apparent. The hook only changes the termination path on allocation failure (already a fatal condition); it doesn't expose new inputs, parse untrusted data, or affect auth/crypto/permissions. The crash report path itself is pre-existing and unchanged.

Level of scrutiny

Moderate-to-high. While the diff is small and well-reasoned, it changes process-global behavior for every Bun build: a new nightly feature gate is enabled crate-wide, and a global hook is installed that runs under memory pressure on the abort path. The PR description argues the hook is allocation-free and that PANIC_STAGE guards re-entry, which looks correct from the code, but this is exactly the kind of subtle, runs-once-per-process, hard-to-debug-if-wrong infrastructure where a maintainer should confirm the design choice (vs. -Zoom=panic or a SIGABRT handler, both of which the description considered and rejected).

Other factors

The change is well-tested: a real Vec::with_capacity(1<<61) exercises the genuine RawVec → handle_alloc_error path end-to-end, with ASAN accommodations. The test-only binding additions follow the established pattern in crash_handler_jsc.rs. No outstanding reviewer comments remain — my prior nit is resolved. Given the global/infrastructural nature of the change rather than its size, I'm deferring rather than auto-approving.

@robobun

robobun commented Jun 2, 2026 •

Copy link
Copy Markdown
Collaborator Author

CI status summary — the diff is green; the remaining red is macOS agent capacity, unrelated to this change.

Two full builds (59786 and its re-roll 59893): every job that obtained an agent passed — 280+ jobs each, all build lanes, all Linux/Windows test lanes. The only failures ever recorded:

  • darwin-14/26 test lanes expiring in the agent queue (0s duration, jobs never executed; repeated over ~8h of macOS pool starvation, auto-retries still queued).
  • When a darwin job did run (build 59786): one runner had a broken Postgres container (role "bun_sql_test" does not exist → ~800 cascading sql.test.ts failures), another has Docker without buildx (unknown flag: --progress → 2 docker-build tests). Both are runner-environment issues that also have nothing to do with this PR — and zero out of memory strings appear in any log, i.e. the new hook never fired outside the test that targets it.
  • test/cli/run/run-crash-handler.test.ts is skip-listed in CI by test/expectations.txt (pre-existing segfault should report flake entry), so the darwin lanes never execute this suite in the first place.

Verification of the change itself was done locally on the ASAN debug build: without src/ changes the new infallibleOutOfMemory case fails (std abort message, exit 134, no report); with them it produces the "Bun has run out of memory" message, a report URL, and a POST to the self-hosted report server — and panic/segfault/outOfMemory behave unchanged.

I've used my one CI re-roll; not pushing further retriggers. Ready for maintainer review/merge.


Update: the re-rolled build's darwin-14-x64 shard landed on the same broken runner (macOS-13-x64-1) and failed identically — 6,236× role "bun_sql_test" does not exist, zero OOM strings, this suite still skip-listed. That agent's Postgres container is poisoning every build it picks up; the lane needs the runner fixed (or a lucky agent draw), not a code change.

Final state of build 59893: darwin-26-aarch64 passed once it drew a healthy agent. darwin-14-x64 failed only on macOS-13-x64-1 (broken Postgres, both attempts). darwin-14-aarch64 ran on darwin-aarch64-14.5-1 and failed only in s3.test.ts — ~110 MinIO tests dying with ConnectionClosed across all 4 in-runner attempts, i.e. that runner's MinIO service is down. Zero out of memory strings in any log. Every failure across both builds traces to a specific broken runner service, never to this diff.

Build 60935 (post-rebase): the debian-13 x64-asan shard fails only in bunx.test.ts "should handle package that requires node 24" (bunx --bun @angular/cli@latest --help exits 3, all 4 attempts). This reproduces identically with the stock release Bun (USE_SYSTEM_BUN=1 bun test ... -t "requires node 24" → exit 3): @angular/cli@latest on the live registry now fails its own node-version self-check, so every lane and branch that draws this test will fail until the test pins a version. Unrelated to this PR; crash-handler suite remains skip-listed and no OOM output appears in any log.

Build 60935 final: the @angular/cli@latest drift now fails the bunx shard on all 13 platform lanes (debian/ubuntu/alpine/windows, x64/aarch64/baseline/asan/musl) — one test, bunx.test.ts "should handle package that requires node 24", every failure, zero OOM output. All 267 other shards passed. Any build of any branch will be red until that test pins a version.

robobun added 3 commits June 5, 2026 23:34
Only explicit Result<_, AllocError> paths reached the crash handler's
CrashReason::OutOfMemory report via bun_core::handle_oom. Ordinary
Vec/Box/String allocation failure went to the global allocator's
handle_alloc_error, whose std default hook prints "memory allocation of
N bytes failed" and aborts with SIGABRT: no OOM message, no trace
string, no crash report.

Install a std::alloc::set_alloc_error_hook in install_hooks() that
routes handle_alloc_error into the existing out_of_memory() path, so
every OOM produces the same "Bun ran out of memory" report. The hook
body is a single diverging call and performs no allocation; a second
OOM inside the report path hits the PANIC_STAGE re-entry guard.

Adds a crash_handler.infallibleOutOfMemory testing binding that fails a
real Vec growth through the global allocator and wires it into the
automatic crash reporter tests. Cargo.lock picks up the bstr dependency
bun_bin gained in #31668.
The inline-capacity hint was a literal 8, which desynced when
infallibleOutOfMemory became the 9th entry. Derive it from the entry
list so it cannot drift.
@robobun
robobun force-pushed the farm/afc126d6/oom-alloc-error-hook branch from 7317ea0 to 3a4863d Compare June 5, 2026 23:38

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/cli/run/fixture-crash.js`:
- Around line 14-15: The invalid-approach branch in fixture-crash.js currently
only prints the usage string and returns success; update that branch so it fails
loudly by calling process.exit with a non-zero code after the console.error call
(i.e., when the approach is unrecognized in the invalid-approach / default
branch that prints "usage: bun fixture-crash.js ..."). Ensure the exit code is
non-zero (e.g., 1) so misconfigured tests do not pass silently.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 20ae8ee1-49ef-4d74-a2e7-e783430e4d1e

📥 Commits

Reviewing files that changed from the base of the PR and between 08226e2 and 3a4863d.

📒 Files selected for processing (6)
  • src/crash_handler/handle_oom.rs
  • src/crash_handler/lib.rs
  • src/js/internal-for-testing.ts
  • src/runtime/api/crash_handler_jsc.rs
  • test/cli/run/fixture-crash.js
  • test/cli/run/run-crash-handler.test.ts

Comment thread test/cli/run/fixture-crash.js
An unrecognized approach printed usage and exited 0, which could mask a
miswired test as a successful run.
@robobun

robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #38888, which installs the same alloc error hook and also removes the dead OOM wrapping around the collection APIs that the hook makes unnecessary (the follow-on cleanup requested by the maintainers). Closing this one in its favor.

@robobun robobun closed this Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant