Skip to content

fs.rm: continue past vanished entries, retry transient errors, report the failing path - #41480

Open
robobun wants to merge 6 commits into
mainfrom
robobun/6b439be5/fs-rm-walker-errors
Open

robobun wants to merge 6 commits into
mainfrom
robobun/6b439be5/fs-rm-walker-errors

Conversation

@robobun

@robobun robobun commented Sep 6, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • fs.rm(dir, { recursive: true }) fails with ENOENT when another process removes a child between getdents64 and unlinkat/openat, or removes a directory the walker still has open (Linux getdents64 then reports ENOENT). With force: true the caller reads that ENOENT as "the root is missing" and returns success with the rest of the tree still on disk (zig_delete_tree in src/runtime/node/node_fs.rs and the FileNotFound check in rm()).
  • maxRetries and retryDelay are parsed but never used. EBUSY, EMFILE, ENFILE, ENOTEMPTY and EPERM fail on the first attempt.
  • The walker reports errors as error-set names. dt_err maps any errno outside its table to Unexpected, which map_anyerror_to_errno* turns into EFAULT. The non-recursive path has a third table with the same EFAULT fallthrough. The reported path is always the root, never the entry that failed.

Fix

  • zig_delete_tree returns sys::Maybe<()>: the raw errno plus the path of the entry that failed (dt_entry_path joins the root with the stack of directory names). ENOENT for a child, and ENOENT from getdents64 on a directory that was removed while open, continue the walk. Only the root can return ENOENT, so the force check in rm() is correct again. The three name tables are gone, the non-recursive unlink reports its raw errno, and the unreachable recursive branch of the native rmdir is removed (the JS layer routes recursive rmdir through rm).
  • rm_with_retries wraps the recursive walk. It retries the errors Node's rimraf retries, maxRetries times, after a sleep of retryDelay * attempt ms. A retry that finds the path gone succeeds, as in Node. The non-recursive unlink is not retried: Node ignores maxRetries when recursive is not true.
  • Both getdents64 iterators (dir_iterator.rs, sys/lib.rs) loop on EINTR. On macOS, dt_delete_file reports ENOENT when the entry vanishes between unlinkat and the lstatat that disambiguates EPERM.
  • Verified: test/js/node/fs/rm-recursive-errors.test.ts (7 tests, all fail on main). Also fs.test.ts, promises.test.js, test-fs-rm.js, and cargo check for darwin and windows targets.

Background

  • zig_delete_tree is the recursive delete walker behind fs.rm with recursive: true. It keeps a stack of open directory descriptors (16 deep) and iterates each with getdents64. Past 16 levels it switches to zig_delete_tree_min_stack_size_with_kind_hint, which keeps one descriptor open and restarts from its root after each directory.
  • Node's async fs.rm is lib/internal/fs/rimraf.js. It ignores ENOENT on a child, retries the whole removal on EBUSY, EMFILE, ENFILE, ENOTEMPTY and EPERM, and reports the path of the child that failed. The sync path (C++ RmSync) sleeps between retries the same way this change does.
  • sys::Error carries errno, the syscall tag and a path. rm() re-tags the walker's error as rm and strips the Windows \\?\ prefix from the path.
Notes

Related open PRs that each cover one face of this: #35800 and #39710 (ENOENT race), #35749 (errno passthrough), #35927 (failing path), #39003 (maxRetries), #41486 (ENOTDIR instead of EFAULT on the non-recursive path, the JS-side lstat rethrow, and an unrelated stream path fix). This change covers their Linux and macOS parts in one walker. Not covered: the Windows STATUS_DELETE_PENDING mapping on the directory open from #39710, and the lstat rethrow and stream path fix from #41486. #41486 owns the removal of map_rm_errno_narrow. This branch needs a rebase once it merges.

The race tests start a worker that deletes the same tree while the main thread walks it. Three tests use a deleter that unlinks from the end of each listing, so the walker sees entries vanish between getdents64 and unlinkat/openat. On the released binary they fail 29 of 30 runs. The fourth test runs two recursive walkers at once: the one that falls behind still holds a directory open when the other removes it, and its next getdents64 reports ENOENT (IS_DEADDIR, verified with a raw syscall(SYS_getdents64) on tmpfs and overlayfs). Without the iterator arm it failed 5 of 6 runs. With the fix the file passed 8 of 8 runs under the debug build.

The EMFILE tests run bun under ulimit -n 64, open descriptors until EMFILE, then free two: enough to open root and root/a, not root/a/b. That gives a deterministic EMFILE at a known entry. The retry test frees descriptors from a timer on the main thread while the walk retries on the pool thread.

Retries sleep on the calling thread. For the async flavours that is a work-pool thread, blocked for at most retryDelay * maxRetries * (maxRetries + 1) / 2 ms. Node's C++ rmSync sleeps the same way. Node's async rimraf uses setTimeout instead.

The EINTR arms have no test: getdents64 does not return EINTR on a local filesystem without syscall injection. They are the Linux and BSD twins of the existing getdirentries64 loop on macOS.

The deep-tree walker (zig_delete_tree_min_stack_size_with_kind_hint) names the depth-16 directory in its error, not the deeper entry, because it does not keep the chain of names it descended through.

Suites run: test/js/node/fs/rm-recursive-errors.test.ts (x8), test/js/node/fs/fs.test.ts, test/js/node/fs/promises.test.js, test/js/node/fs/dir.test.ts, test/js/node/test/parallel/test-fs-rm.js, test-fs-rmdir-throws-on-file.js, bun scripts/rust-check-all.ts aarch64-apple-darwin x86_64-pc-windows-msvc.


no test proof · iteration 4 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/node/fs/rm-recursive-errors.test.ts

… the failing path

The recursive delete walker returned ENOENT for a child that another
process removed first, and for a directory that was removed while the
walker still had it open (getdents64 reports ENOENT for it). With
force: true the caller read that as a missing root and returned success
with the rest of the tree in place. The walker now skips a vanished
child, treats a removed directory as finished, and only returns ENOENT
for the root.

maxRetries and retryDelay were parsed but never used. The removal is
now retried on EBUSY, EMFILE, ENFILE, ENOTEMPTY and EPERM, the set
Node's rimraf retries, with a delay of retryDelay * attempt. A retry
that finds the path gone succeeds.

Errors carry the raw errno and the path of the entry that failed
instead of a name string that mapped unknown errors to EFAULT and
always named the root. The non-recursive unlink path reports its raw
errno too.

The recursive branch of the native rmdir is unreachable: the JS layer
routes recursive rmdir through rm. It is removed.

getdents64 is retried on EINTR in both directory iterators.
@coderabbitai

coderabbitai Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Essentials

Run ID: f61fd096-e0b7-4042-b1ce-877f0876a9e2

📥 Commits

Reviewing files that changed from the base of the PR and between f42e980 and 494381d.

📒 Files selected for processing (4)
  • src/runtime/node/dir_iterator.rs
  • src/runtime/node/node_fs.rs
  • src/sys/lib.rs
  • test/js/node/fs/rm-recursive-errors.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 0 remain after this review.


Walkthrough

Changes

Recursive removal reliability

Layer / File(s) Summary
Interrupted directory iteration retries
src/runtime/node/dir_iterator.rs, src/sys/lib.rs
Linux, Android, and FreeBSD directory reads retry after EINTR.
rm retry and error construction
src/runtime/node/node_fs.rs
rm operations use retry rules and report errno, syscall names, and affected paths.
Recursive delete traversal updates
src/runtime/node/node_fs.rs
Recursive deletion uses sys::Maybe, tolerates concurrent ENOENT, and annotates traversal errors with full paths.
Concurrent deletion and EMFILE validation
test/js/node/fs/rm-recursive-errors.test.ts
Tests cover concurrent removal, force modes, simultaneous walkers, nested error paths, and EMFILE retries.

Suggested reviewers: dylan-conway, jarred-sumner

Merge Risk: ⚪ Minimal · up to 49438

The change improves recursive removal under races and transient errors, with targeted coverage for concurrent deletion, retries, and error paths. No actionable merge-blocking risk is currently established.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the primary changes: handling vanished entries, retrying transient errors, and reporting the failing path.
Description check ✅ Passed The description explains the problem, implementation, scope, limitations, and verification steps. It does not use the exact template headings, but it provides the required information in equivalent se…

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the claude label Sep 6, 2026
@robobun

robobun commented Sep 6, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 1:40 AM PT - Sep 6th, 2026

❌ @robobun, your commit 49e909c has 1 failures in Build #110809 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 41480

That installs a local version of the PR into your bun-41480 executable, so you can run:

bun-41480 --bun

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

  • 🟡 src/runtime/node/node_fs.rs — nit: the comment "See the matching comment in rmdir" now points at nothing — this PR deleted the recursive branch (and its Windows path-resolution comment) from rmdir(). Fix: make the comment self-contained (it already explains the Windows rooted-but-driveless case) and drop the cross-reference.

    Extended reasoning...

    The referenced comment block in rmdir() ("On Windows a rooted-but-driveless path …") was removed at lines 7431–7452 of the diff. The remaining rm() comment still tells readers to look there, which will send the next maintainer on a dead-end search. No runtime effect; documentation-only.

    Verification: nit — The comment at src/runtime/node/node_fs.rs:7454 reads "See the matching comment in rmdir: pre-resolve the path on Windows so rooted-but-driveless paths …". This PR deletes the referenced comment: the diff hunk @@ -7458,28 +7431,6 @@ removes the entire if args.recursive { … } block from rmdir(), including the "On Windows a rooted-but-driveless path ("/tmp/foo") must resolve…

Comment thread test/js/node/fs/rm-recursive-errors.test.ts Outdated
Comment thread test/js/node/fs/rm-recursive-errors.test.ts Outdated
Comment thread src/runtime/node/node_fs.rs Outdated
Comment on lines +7454 to +7459
// On Windows a rooted-but-driveless path ("/tmp/foo") must resolve
// against the cwd drive. The dt_* helpers go through
// Syscall::*at -> to_nt_path / normalize_path_windows, which do not
// add the drive and would turn "/tmp/foo" into a nonexistent NT
// name (ENOENT). Pre-resolve with slice_z so the path carries a
// drive letter, the same way existsSync/statSync/unlinkSync see it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

Comment on lines +7467 to +7468
// The walker swallows ENOENT below the root, so this is the
// root itself.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

Comment thread src/runtime/node/node_fs.rs Outdated
Comment on lines +9194 to +9197
// zig_delete_tree — recursive delete-tree. An error names the entry that
// failed (`sys::Error::path`) and carries the raw errno. ENOENT below the
// root is not an error: another process removed the entry first. ENOENT
// for the root itself is returned, which `fs.rm` with `force: false` needs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

Comment thread src/runtime/node/node_fs.rs Outdated
Comment on lines +9200 to +9202
// Implemented on top of `bun_sys` primitives (`openat` + `unlinkat`). The
// structure: 16-slot stack, treat_as_dir flip-flop, close-then-deleteDir,
// retry-on-DirNotEmpty.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

@robobun

robobun commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed in 939e655:

  • The rm() comment no longer points at the removed rmdir branch. It explains the Windows drive resolution on its own.
  • Every Worker in the test file has its error event wired to the awaited promise. The main thread waits for the second walker with a bounded Atomics.wait instead of a spin.
  • In the two-walker test a root ENOENT from either side counts as success. That is the outcome when one walker is stalled until the other has removed the root, which is not the bug under test. The other race tests never remove the root from the deleter, so they are not exposed to that window.

Comment thread src/runtime/node/node_fs.rs Outdated
Comment on lines +9204 to +9207
/// Runs `remove` up to `maxRetries + 1` times. A retry happens for the
/// errors Node's rimraf retries (EBUSY, EMFILE, ENFILE, ENOTEMPTY, EPERM)
/// after a sleep of `retryDelay * attempt` milliseconds. A retry that finds
/// the path gone is a success: the first attempt proved the path existed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

Comment thread src/runtime/node/node_fs.rs Outdated
Comment on lines +9234 to +9236
/// The path of the entry the walker is working on: `sub_path`, then the name
/// of each directory entered below it (the stack items that own a name),
/// then `name` when given.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

Comment thread src/runtime/node/node_fs.rs Outdated
Comment on lines +9384 to +9385
// Linux reports ENOENT from getdents64 on a directory that was
// removed while open. Nothing is left to delete in it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

Comment thread src/runtime/node/node_fs.rs Outdated
Comment on lines +9666 to +9669
/// Deletes `sub_path` (relative to `self_`) with one open directory at a
/// time. The walker above switches to this once its stack is full. An error
/// names `sub_path`, not the deeper entry that failed: this function does
/// not keep the chain of directory names it descended through.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

@robobun

robobun commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

2078765 shortens the comments the comment linter flagged in node_fs.rs: the Windows path note in rm(), the walker banner, the rm_with_retries and dt_entry_path docs, the iterator ENOENT arms, and the deep-tree walker doc. No code change.

Comment thread src/runtime/node/node_fs.rs Outdated
Comment on lines +7454 to +7455
// Syscall::*at does not add the cwd drive to a rooted path
// ("/tmp/foo"); slice_z does.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

Comment thread src/runtime/node/node_fs.rs Outdated
Comment on lines +9190 to +9191
// zig_delete_tree — recursive delete-tree. Errors carry the raw errno and the
// path of the entry that failed. ENOENT is only returned for the root.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

Comment thread src/runtime/node/node_fs.rs Outdated
Comment on lines +9194 to +9195
/// Retries `remove` on the errors Node's rimraf retries, `maxRetries` times,
/// sleeping `retryDelay * attempt` ms. A retry that finds the path gone succeeds.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

Comment thread src/runtime/node/node_fs.rs Outdated
Comment on lines +9651 to +9652
/// One open directory at a time, for trees deeper than the stack above.
/// Errors name `sub_path`: the chain of names below it is not kept.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

@robobun

robobun commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

494381d reduces the remaining flagged comments to one line each (the Windows path note in rm(), the walker banner, the rm_with_retries doc, and the deep-tree walker doc). No code change.

Comment thread src/runtime/node/node_fs.rs Outdated
Node ignores maxRetries and retryDelay when recursive is not true.
@robobun

robobun commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Fixed in 64df19b: the non-recursive unlink is no longer wrapped in rm_with_retries. Node documents maxRetries and retryDelay as ignored when recursive is not true, so a non-recursive rmSync on an immutable file throws EPERM at once again. The recursive path keeps the retries.

@robobun

robobun commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Overlap note. Three open PRs removed map_rm_errno_narrow from src/runtime/node/node_fs.rs: this one, #33436, and #41486. #41486 now owns that change (the non-recursive unlink errno passthrough and the JS-side lstat rethrow). #33436 is closed in its favor.

What this means for this PR:

The body's reference to #33436 is now stale. The force handling of an ENOTDIR ancestor that it mentions is not wanted: Node's fs.promises.rm and callback fs.rm throw ENOTDIR for { recursive: true, force: true } on a path through a regular file. Only Node's rmSync succeeds there.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The non-recursive unlink now bypasses rm_with_retries (commit 64df19b), so maxRetries/retryDelay are ignored when recursive is false as Node documents — that addresses the earlier blocking concern. I re-reviewed the walker and retry changes and found no further issues; given this rewrites ~300 lines of the delete-tree walker's error propagation across Linux/macOS/Windows branches, a human pass on the cross-platform paths is still worthwhile.

What was reviewed:

  • rm_with_retries is now reached only from the args.recursive branch; the non-recursive path calls sys::unlink once and surfaces the raw errno.
  • ENOENT swallowing is scoped to child entries (unlinkat/openat/getdents/min-stack recursion) — the root's ENOENT still propagates so force semantics stay correct.
  • dt_entry_path filters name_is_borrowed so the root isn't duplicated in the reported path; the popped top is passed via the name argument since it's already off the stack.
  • EINTR loops in both getdents iterators mirror the existing macOS getdirentries64 pattern.
Extended reasoning...

Overview

This PR rewrites error propagation in the recursive fs.rm delete-tree walker (src/runtime/node/node_fs.rs), replacing three string-name→errno mapping tables with direct sys::Maybe<()> propagation carrying the raw errno, syscall tag, and the failing entry's path. It adds ENOENT-tolerance for child entries removed concurrently, a rm_with_retries wrapper implementing Node's maxRetries/retryDelay semantics, EINTR retry loops in both getdents iterators, and a new race/EMFILE test file. The unreachable recursive branch of native rmdir() is deleted (JS routes recursive rmdir through rm).

Since the last review, commit 64df19b reverted the non-recursive rm path from rm_with_retries(args, || sys::unlink(dest)) back to a single sys::unlink(dest) call, addressing the Node-compat regression flagged earlier (Node ignores maxRetries/retryDelay when recursive is not true). Commit 494381d shortened comments. Commit 939e655 wired worker error events to reject the awaited promises and made the two-walker race test tolerate a root-ENOENT loser, addressing both earlier 🟡 test-robustness comments.

Security risks

No new attack surface. The change narrows error handling (raw errno passthrough replaces a lossy table with an EFAULT fallthrough) and adds bounded retries with a caller-controlled sleep on a work-pool thread. Paths reported in errors are constructed from directory-entry names read from getdents — no user-controlled string concatenation into a syscall argument. The rm_with_retries sleep is retry_delay * attempt with u64::from widening before the multiply, so no overflow on the documented value ranges.

Level of scrutiny

High. This is a ~300-line rewrite of a Node-compat filesystem walker with per-OS #[cfg] branches (macOS EPERM disambiguation, Windows path pre-resolution), a change in error-return type rippling through four functions, and new concurrency-tolerant behavior. The PR description notes Windows STATUS_DELETE_PENDING and force+ENOTDIR-ancestor handling are explicitly out of scope. .claude/docs/landing-prs.md calls for auditing every sibling platform backend and citing Node source for magic constants — the retry errno set matches Node's lib/internal/fs/rimraf.js, but a maintainer should confirm the Windows walker path (which this PR touches only via the slice_z pre-resolution comment) still behaves.

Other factors

The new test file spawns worker threads with a SharedArrayBuffer barrier and a ulimit -n 64 subprocess for deterministic EMFILE — the PR notes 8/8 debug-build passes and 29/30 failure rate on the released binary for the race tests, which is reasonable evidence they exercise the fix. The github-actions[bot] inline comments on node_fs.rs (likely formatting/lint) were followed by fix commits. No CHANGES_REQUESTED reviews from human reviewers are outstanding. The exit reason was dry_streak, so the hunt ran to completion. Given the scope and cross-platform surface, deferring rather than approving.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

@robobun

robobun commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

CI state at 49e909c: 180 of 181 jobs pass. The one red job is test/js/node/test/parallel/test-cluster-primary-error.js (a timeout on debian 13 x64-asan) which this diff does not touch. It has been reported separately. test/js/node/fs/rm-recursive-errors.test.ts passes on every lane. The earlier runs on this PR failed only on unrelated tests (local-sql, serve-pending-promise-abort-leak, fetch-backpressure, the napi test_object stall), also reported. Ready for review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants