Skip to content

test: surface ASAN status to leak fixtures via bunEnv - #35081

Open
robobun wants to merge 3 commits into
mainfrom
farm/7fa7f8e8/fix-asan-detection-in-leak-fixtures
Open

robobun wants to merge 3 commits into
mainfrom
farm/7fa7f8e8/fix-asan-detection-in-leak-fixtures

Conversation

@robobun

@robobun robobun commented Jul 22, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Eleven leak-test fixtures detect ASAN with process.execPath.includes("bun-asan"). That is true on CI's ASAN lane (the binary is literally named bun-asan) but false for a local bun bd build, which is ASAN-instrumented but named bun-debug. So the fixtures apply the tight non-ASAN RSS threshold locally and report false leaks.

Repro on main:

$ bun bd test test/js/web/timers/setTimeout.test.js -t "doesn't leak when"
(fail) setTimeout doesn't leak when clear is called inside its own callback
  error: Memory leak detected: RSS grew by 139.7 MB
(fail) setTimeout doesn't leak when refresh is called inside its own callback
(fail) setTimeout doesn't leak when repeat is called inside its own callback
 0 pass / 3 fail

The 10 MB cap is compared against the ~138 MB ASAN-quarantine growth because isASAN is false; the 192 MB ASAN cap is never selected. CI does not see this because bun-asan matches the name check.

Fix

test/harness.ts already detects ASAN correctly via isASANEnabled() from bun:internal-for-testing (a compile-time #if ASAN_ENABLED probe), and every one of these fixtures is spawned with env: bunEnv (either directly or via the toRun matcher). So set bunEnv.BUN_TEST_IS_ASAN = "1" once inside harness.ts's existing if (isASAN) block, and have each fixture read process.env.BUN_TEST_IS_ASAN instead of re-detecting.

This is the second lockstep edit of the same detection snippet across these eleven files (fba43af684 added the name check to each); centralizing it in bunEnv means there isn't a third. The env var is runtime-agnostic (works under Node for the dual-runtime fixtures) and survives the tempdir-copy pattern used by bun-write-leak.test.ts.

The change only widens the RSS threshold on a binary the harness has confirmed is ASAN-instrumented, so a real leak still trips the tight threshold on release builds. CI behaviour is unchanged: bun-asan was already detected before, and still is.

Verification

Under ./build/debug/bun-debug:

execPath.includes("bun-asan") -> false   (old check)
harness.ts detectASAN()        -> true   (now surfaced as BUN_TEST_IS_ASAN=1)

bun bd test test/js/web/timers/setTimeout.test.js -t "doesn't leak when" now passes (3/3).

Files touched (all with the identical one-line env read):

  • test/harness.ts (+BUN_TEST_IS_ASAN inside the existing if (isASAN) block)
  • test/js/web/timers/setTimeout-clear-in-callback-leak-fixture.js
  • test/js/web/timers/setInterval-leak-fixture.js
  • test/js/bun/http/server-fetch-string-leak-fixture.js
  • test/js/bun/io/bun-write-leak-fixture.js
  • test/js/web/fetch/fetch-leak-test-fixture-2.js
  • test/js/node/url/pathToFileURL-leak-fixture.js
  • test/js/node/http2/node-http2-memory-leak.js
  • test/cli/run/esm-bug-leak-fixture.mjs
  • test/cli/run/require-cache-bug-leak-fixture.js
  • test/cli/run/cjs-fixture-leak-small.js
  • test/cli/run/esm-fixture-leak-small.mjs

test/js/node/async_hooks/async-context/async-context-fs-watch.js is intentionally excluded: it is a skip guard (process.exit(0)) rather than an RSS threshold, and the fixture already passes under bun bd, so widening the skip there would hide a passing test.

This is a test-infrastructure fix with no src/ change.


no test proof · iteration 0 · docs-only change; test-proof not applicable

…y name

A local `bun bd` debug build is ASAN-instrumented but the binary is named
`bun-debug`, not `bun-asan`, so `process.execPath.includes("bun-asan")`
returned false and the fixtures applied the tight non-ASAN RSS threshold. Under
`bun bd test test/js/web/timers/setTimeout.test.js` the three "doesn't leak
when clear/refresh/repeat" tests failed with "RSS grew by ~138 MB" against a
10 MB cap. CI was unaffected because its ASAN lane binary is literally named
`bun-asan`.

Probe the runtime the same way harness.ts already does, via
`require("bun:internal-for-testing").isASANEnabled()`, with the name check
as a fallback for Node and for release binaries without the internal testing
flag.
@robobun

robobun commented Jul 22, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 2:50 AM PT - Jul 22nd, 2026

✅ @robobun, your commit ba041e5b152ccb3ee055c038a05b1285da46ad99 passed in Build #77545! 🎉


🧪   To try this PR locally:

bunx bun-pr 35081

That installs a local version of the PR into your bun-35081 executable, so you can run:

bun-35081 --bun

@robobun

robobun commented Jul 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

Reproduced on main with:

bun bd test test/js/web/timers/setTimeout.test.js -t "doesn't leak when"

3 fail before ("RSS grew by ~138 MB" vs a 10 MB cap), 3 pass after.

Test-only change; no src/ diff. After review feedback, detection is centralized: harness.ts sets BUN_TEST_IS_ASAN in bunEnv, and each fixture reads the env var instead of re-probing.

CI: the remaining failures on build 77545 are all retry-flaky and unrelated to this diff (complex-workspace.test.ts install cascade, no-orphans.test.ts PTY timing, bun-install-registry.test.ts Windows hoisting, webview-chrome.test.ts animation, spawn.test.ts Windows timeout). None of the eleven touched fixtures or their driving test files appear, and on non-ASAN lanes the harness change is a no-op. Ready for a maintainer to merge.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Changes

ASAN detection across leak fixtures

Layer / File(s) Summary
Harness ASAN propagation
test/harness.ts
The harness sets BUN_TEST_IS_ASAN for ASAN-enabled runs.
Leak fixture threshold detection
test/cli/run/*-leak-*, test/js/bun/**/*.js, test/js/node/**/*-leak*.js, test/js/web/**/*-leak*.js
Leak fixtures use BUN_TEST_IS_ASAN instead of executable-path checks while retaining ASAN-specific memory thresholds and iteration behavior.

Possibly related PRs

Suggested reviewers: jarred-sumner

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is concise and accurately summarizes the main change: exposing ASAN status to leak fixtures via bunEnv.
Description check ✅ Passed The description covers the problem, fix, and verification details, matching the template's required intent despite different headings.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/cli/run/cjs-fixture-leak-small.js`:
- Line 10: Replace the ASAN fallback in the leak-detection checks with a
basename-only executable-path check, matching the established behavior in
test/harness.ts. Apply this change at test/cli/run/cjs-fixture-leak-small.js:10,
test/cli/run/esm-bug-leak-fixture.mjs:12,
test/cli/run/esm-fixture-leak-small.mjs:12,
test/cli/run/require-cache-bug-leak-fixture.js:10,
test/js/bun/http/server-fetch-string-leak-fixture.js:12,
test/js/bun/io/bun-write-leak-fixture.js:11,
test/js/node/http2/node-http2-memory-leak.js:12,
test/js/node/url/pathToFileURL-leak-fixture.js:11,
test/js/web/fetch/fetch-leak-test-fixture-2.js:18, and
test/js/web/timers/setInterval-leak-fixture.js:12, using the existing basename
utility rather than checking the full process.execPath.

In `@test/js/web/timers/setTimeout-clear-in-callback-leak-fixture.js`:
- Line 20: Update the fallback executable check in the timer leak fixture to
compare against the basename of process.execPath, matching the approach used by
test/harness.ts, so parent-directory names cannot trigger a false match.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2f76a6f2-a8ef-460a-b513-018456ba2e7b

📥 Commits

Reviewing files that changed from the base of the PR and between 12b30c3 and 3f0260e.

📒 Files selected for processing (11)
  • test/cli/run/cjs-fixture-leak-small.js
  • test/cli/run/esm-bug-leak-fixture.mjs
  • test/cli/run/esm-fixture-leak-small.mjs
  • test/cli/run/require-cache-bug-leak-fixture.js
  • test/js/bun/http/server-fetch-string-leak-fixture.js
  • test/js/bun/io/bun-write-leak-fixture.js
  • test/js/node/http2/node-http2-memory-leak.js
  • test/js/node/url/pathToFileURL-leak-fixture.js
  • test/js/web/fetch/fetch-leak-test-fixture-2.js
  • test/js/web/timers/setInterval-leak-fixture.js
  • test/js/web/timers/setTimeout-clear-in-callback-leak-fixture.js

Comment thread test/cli/run/cjs-fixture-leak-small.js Outdated
Comment thread test/js/web/timers/setTimeout-clear-in-callback-leak-fixture.js Outdated

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — mechanical test-fixture change that mirrors the isASAN probe already in test/harness.ts. The inline nit about async-context-fs-watch.js is optional (skip guard, not an RSS threshold — arguably outside the PR's stated scope).

Extended reasoning...

Overview

Eleven leak-test fixture files switch their isASAN detection from process.execPath.includes("bun-asan") to a try/catch around require("bun:internal-for-testing").isASANEnabled(), keeping the name check as a fallback. No src/ changes, no threshold values changed, no assertions weakened. The identical IIFE is applied 11 times.

Correctness checks

  • isASANEnabled exists: src/js/internal-for-testing.ts:387 → jsFunction_isASANEnabled in src/jsc/bindings/InternalForTesting.cpp:66. It's the same probe test/harness.ts:58-61 already uses, so the "same as harness.ts" comment is accurate.
  • .mjs fixtures: both already do const require = createRequire(import.meta.url) before the new IIFE, so require("bun:internal-for-testing") is defined.
  • .js fixtures with top-level import (pathToFileURL-leak-fixture.js, node-http2-memory-leak.js, fetch-leak-test-fixture-2.js): Bun allows mixed import/require in .js, and several of these files already use require elsewhere.
  • Not a silent test weakening: on non-ASAN release builds, isASANEnabled() returns false (or the require throws → falls through to name check → false), so the tight threshold is still selected. Only ASAN-instrumented binaries get the wide threshold — which was the intent of the original code. CI behavior is unchanged in both lanes.
  • Same-class sweep: grepped for remaining bun-asan name checks in test/; the only fixture-level site left is async-context-fs-watch.js:5 (.endsWith variant), which is a skip guard rather than an RSS threshold. Flagged as a nit inline. All other hits are comments or .test.ts files that import isASAN from harness.

Security risks

None — test-fixture-only change, no user-facing surface.

Level of scrutiny

Low. Test infrastructure only, mechanical pattern replication of an established harness helper, no threshold or assertion changes.

Other factors

The one inline finding is an optional same-class nit outside the PR's declared scope; it doesn't block.

Comment thread test/cli/run/cjs-fixture-leak-small.js Outdated

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — mechanical test-fixture change, identical 7-line probe applied to all 11 sites.

What was reviewed:

  • Confirmed the IIFE matches test/harness.ts's detectASAN() and that isASANEnabled() exists as the referenced probe.
  • Checked require() is in scope at each site: the two .mjs fixtures createRequire above the IIFE; the ESM-syntax .js fixtures are Bun-only where require is a global (several already call require("bun:jsc") later).
  • Prior nit about async-context-fs-watch.js was answered (intentional exclusion — widening that skip would hide a passing test); CodeRabbit's basename-fallback note was also resolved (the fallback is dead under bunEnv).
Extended reasoning...

Overview

Test-infrastructure-only change touching 11 leak-test fixture .js/.mjs files under test/. Each file replaces const isASAN = process.execPath.includes("bun-asan") with an IIFE that first tries require("bun:internal-for-testing").isASANEnabled() and falls back to the pre-existing name check. No src/ changes; RSS thresholds themselves are unchanged. The pattern is copied verbatim from test/harness.ts:56-62 (verified).

Security risks

None. Test fixtures only; no user-facing code, no auth/crypto/permissions, no new inputs parsed.

Level of scrutiny

Low. This is a mechanical sweep of an identical snippet across 11 files whose only effect is selecting which of two pre-existing RSS thresholds applies. The change can only widen the threshold on binaries that are actually ASAN-instrumented (per the #if ASAN_ENABLED probe), so it cannot mask a real leak on release builds and does not alter CI behavior (CI's bun-asan binary already matched the old check). Worst case is a fixture that previously false-failed under bun bd now passes locally — which is the stated goal.

Other factors

  • The two .mjs fixtures bind require via createRequire(import.meta.url) before the new IIFE, so the require("bun:internal-for-testing") call resolves. The remaining ESM-syntax fixtures (node-http2-memory-leak.js, pathToFileURL-leak-fixture.js, fetch-leak-test-fixture-2.js, etc.) are Bun-only and already rely on Bun's global require in ESM elsewhere in the same file, so no new assumption is introduced.
  • The try/catch means Node.js dual-runtime fixtures fall through to the name check (correctly false under Node), preserving prior behavior.
  • All three review threads (CodeRabbit ×2 on basename fallback, my prior note on the .endsWith sibling in async-context-fs-watch.js) are resolved with stated rationale; nothing outstanding.
  • PR description includes a before/after repro (bun bd test setTimeout.test.js -t "doesn't leak when": 3 fail → 3 pass).

Review feedback: copy-pasting the detection IIFE into 11 fixtures is
lockstep-edit #2 of the same snippet. harness.ts already knows whether the
running binary is ASAN-instrumented (detectASAN()), and every one of these
fixtures is spawned with env: bunEnv, so set BUN_TEST_IS_ASAN there once and
have each fixture read process.env.BUN_TEST_IS_ASAN. One line per fixture,
runtime-agnostic, survives tempdir copying, single source of truth.
@robobun robobun changed the title test: detect ASAN via isASANEnabled() in leak fixtures, not the binary name test: surface ASAN status to leak fixtures via bunEnv Jul 22, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The revised env-var approach (BUN_TEST_IS_ASAN set in bunEnv) is cleaner than the per-fixture probe — single source of truth in harness.ts's detectASAN(), no bun:internal-for-testing boilerplate, and it moots the basename-fallback concern entirely.

What was reviewed:

  • Verified all 11 fixture spawn sites propagate bunEnv (either env: bunEnv, {...bunEnv, …}, or the .toRun() matcher which uses bunEnv), so BUN_TEST_IS_ASAN reaches every fixture and there's no regression on CI's bun-asan lane vs. the old name check.
  • Confirmed the new key survives the BUN_DEBUG_* deletion loop in harness.ts and that ciEnv (snapshotted before it's set) is only used for buildkite-agent secret get.
  • The earlier sibling-site note (async-context-fs-watch.js) was resolved as an intentional exclusion.
Extended reasoning...

Overview

Test-infrastructure-only change: 11 leak fixtures previously detected ASAN via process.execPath.includes("bun-asan"), which is false for local bun bd builds (ASAN-instrumented but named bun-debug), causing false-positive leak failures locally. The final revision has test/harness.ts set bunEnv.BUN_TEST_IS_ASAN = "1" inside its existing if (isASAN) block, and each fixture reads process.env.BUN_TEST_IS_ASAN === "1". No src/ changes.

The PR went through two iterations — first a per-fixture require("bun:internal-for-testing").isASANEnabled() probe with a name-check fallback, then (commit ba041e5) the current env-var approach. The PR description is stale and still describes the first iteration, but the diff and commit message are clear.

Security risks

None. Test-only, no runtime code paths, no external input handling.

Level of scrutiny

Low. Mechanical replacement of a boolean detection expression across 11 fixture files plus a 4-line addition to bunEnv. The only correctness question is whether every fixture's spawn site actually passes bunEnv — otherwise the env var would be absent and the ASAN threshold would not be selected on CI's bun-asan lane (a regression vs. the old name check). I traced each fixture to its test driver: setTimeout.test.js, setInterval.test.js (via .toRun()), bun-write-leak.test.ts (via .toRun()), s3.leak.test.ts (...bunEnv — but that spawns a different s3-local fixture, not the io one), fetch-leak.test.ts (...bunEnv), pathToFileURL.test.ts (via .toRun()), node-http2.test.js (...bunEnv), leaks-test.test.ts (via .toRun()), and require-cache.test.ts (all four cli/run fixtures, env: bunEnv). The .toRun() matcher in harness.ts uses env: bunEnv. All covered.

Other factors

  • CodeRabbit's basename concern on the earlier revision is moot: the name-check fallback is gone.
  • My earlier note about async-context-fs-watch.js (a same-class .endsWith("bun-asan") skip guard) was answered: intentionally excluded because widening it would hide a locally-passing test, and it's a TODO skip rather than a threshold selector. Reasonable.
  • bunEnv spreads process.env, so a nested test spawn would inherit the flag — correct, since a child of an ASAN binary is the same ASAN binary.
  • The Node dual-runtime fixtures correctly get isASAN = false under Node (env var unset), which is right since Node isn't ASAN-instrumented.

@robobun

robobun commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Still reproducible on current main (e697804), this time through test/js/bun/io/bun-write-leak.test.ts:

$ bun bd test test/js/bun/io/bun-write-leak.test.ts
error: Memory usage is too high
stdout: Memory usage: 335 MB
(fail) Bun.write should not leak the output data

bun-debug reports isASANEnabled() === true and starts at ~335 MB RSS, so that fixture's 256 MB non-ASAN cap trips on the first iteration. With this PR's test/harness.ts and test/js/bun/io/bun-write-leak-fixture.js hunks applied on top of today's main (both still apply cleanly) the test passes: peak RSS across the 300 iterations is ~631 MB, under the 768 MB ASAN cap, stable across 5 runs.

The branch is 611 commits behind main and 7 of the other fixtures now conflict, in every case because main added the rss() / Bun.unsafe.memoryFootprint helper right next to the isASAN line this PR replaces:

  • test/cli/run/cjs-fixture-leak-small.js
  • test/cli/run/esm-bug-leak-fixture.mjs
  • test/cli/run/esm-fixture-leak-small.mjs
  • test/cli/run/require-cache-bug-leak-fixture.js
  • test/js/bun/http/server-fetch-string-leak-fixture.js
  • test/js/node/url/pathToFileURL-leak-fixture.js
  • test/js/web/timers/setTimeout-clear-in-callback-leak-fixture.js

Resolving each one means keeping main's rss declaration and taking this branch's isASAN line. Not opening a separate PR for the bun-write-leak fixture since this one already covers it.

@robobun

robobun commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Still reproducible on current main (4448a2e) through the repro in the PR description. A local bun bd build (ASAN on) fails all three tests:

$ bun bd test test/js/web/timers/setTimeout.test.js -t "doesn't leak when"
error: Memory leak detected: RSS grew by 142.5 MB
(fail) setTimeout doesn't leak when clear is called inside its own callback
(fail) setTimeout doesn't leak when refresh is called inside its own callback
(fail) setTimeout doesn't leak when repeat is called inside its own callback
 0 pass / 3 fail

setTimeout-clear-in-callback-leak-fixture.js still reads process.execPath.includes("bun-asan") on main, so the 142 MB quarantine growth is compared against the 10 MB bound. This PR fixes it. It only needs the rebase described in the previous comment. I am not opening a separate PR for the setTimeout fixture.

@robobun

robobun commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

Still present on main at 0823e50: all eleven fixtures keep the process.execPath.includes("bun-asan") check, and bun bd test test/cli/run/require-cache.test.ts fails locally with leaked: "72 MB" against the 48 MB non-ASAN bound of cjs-fixture-leak-small.js.

This branch now conflicts with main in seven of the fixtures. #36429 (merged Jul 30) added an rss helper on the lines next to the isASAN line in each of them. The conflicts are adjacent-line only. A rebase that keeps both the rss helper and the BUN_TEST_IS_ASAN read resolves them.

@robobun

robobun commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

#41207 rewrites four of the fixtures that this PR changes: cjs-fixture-leak-small.js, esm-bug-leak-fixture.mjs, esm-fixture-leak-small.mjs, and require-cache-bug-leak-fixture.js. After that change, those fixtures have no isASAN line. leak-metric.cjs reads the ASAN status from bun:internal-for-testing. If #41207 lands first, drop the hunks for those four files. The other files in this PR are not affected.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant