Skip to content

feat(core): implement non-recursive watcher to avoid unnecessary watchers being registered#34345

Closed
MaxKless wants to merge 1 commit into
masterfrom
improve-watcher-count
Closed

feat(core): implement non-recursive watcher to avoid unnecessary watchers being registered#34345
MaxKless wants to merge 1 commit into
masterfrom
improve-watcher-count

Conversation

@MaxKless

@MaxKless MaxKless commented Feb 5, 2026

Copy link
Copy Markdown
Contributor

Current Behavior

The Nx native file watcher uses watchexec (v3.0.1) which internally calls notify::INotifyWatcher with RecursiveMode::Recursive. This walks every subdirectory in the
workspace and registers an OS-level watch on each one. The ignore patterns (node_modules/, .git/, .nx/, etc.) are only applied as event filters via the watchexec Filterer
trait — they don't prevent the watch registration itself.

In practice, a typical node_modules contains 30-50k+ subdirectories. All of these get watches registered, even though every event from them is immediately filtered out and
discarded. This causes two problems:

  1. Linux (inotify): Blows past the default max_user_watches limit (usually 8192-65536), causing the watcher to fail with ENOSPC: System limit for number of file watchers
    reached.
  2. All platforms: Wastes OS resources tracking events for tens of thousands of directories whose events will never be used.

On the Nx repo itself: 59,568 directories get watches, of which 46,086 are ignored (node_modules, .git, .nx).

Expected Behavior

The watcher only registers watches for directories that aren't ignored. Using the Nx repo as a benchmark:

  • Before: ~59,568 watches (every directory)
  • After: ~13,482 watches (only non-ignored directories)
  • Reduction: 77%

For typical user projects with larger node_modules trees, the reduction will be even greater.

The NAPI API surface is unchanged — TypeScript consumers see no difference.

Approach

Upgrade watchexec from 3.0.1 to 4.1.0, which introduces WatchedPath::non_recursive(). Instead of passing a single recursive root path to watchexec, we walk the directory
tree ourselves using ignore::WalkBuilder (the same crate and pattern already used in walker.rs) and pass each non-ignored directory as a WatchedPath::non_recursive().

Key design decisions

  1. All platforms, not just Linux. While inotify's hard max_user_watches limit makes Linux the most acute case, registering watches for 50k+ ignored directories is wasteful
    everywhere. Same code path on all platforms keeps things simple.
  2. walk_directories() reuses the walker.rs pattern. Uses ignore::WalkBuilder with the same gitignore/nxignore/glob handling as the existing file walker —
    parent_gitignore_files() for git boundary detection, .nxignore support, and filter_entry for the watcher's static ignore globs.
  3. Dynamic directory creation handling. With non-recursive watches, newly created directories aren't automatically watched. The on_action callback detects
    CreateKind::Folder events and re-walks the workspace to update the watch set. This required removing the #[cfg(target_os = "linux")] guard on directory events in the
    filterer — previously directory events were only allowed through on Linux (for inotify race condition handling), but now all platforms need them to detect new directories.
  4. Weak to avoid circular references. The on_action closure needs access to watchexec.config.pathset() to update watches. Using Arc::downgrade() avoids a
    circular Arc reference (Watchexec → Config → on_action closure → Arc) that would prevent cleanup.
  5. Full re-walk on directory creation rather than incremental updates. config.pathset() replaces the entire path set (no "add" API). Rather than maintaining a separate
    tracking structure, we re-walk the workspace on directory creation. This is simple, correct, and fast since WalkBuilder skips ignored directories.

@nx-cloud

nx-cloud Bot commented Feb 5, 2026

Copy link
Copy Markdown
Contributor

View your CI Pipeline Execution ↗ for commit 588c365

Command Status Duration Result
nx affected --targets=lint,test,test-kt,build,e... ✅ Succeeded 53m 5s View ↗
nx run-many -t check-imports check-lock-files c... ✅ Succeeded 2m 52s View ↗
nx-cloud record -- nx-cloud conformance:check ✅ Succeeded 10s View ↗
nx-cloud record -- nx sync:check ✅ Succeeded <1s View ↗
nx-cloud record -- nx format:check ✅ Succeeded <1s View ↗

☁️ Nx Cloud last updated this comment at 2026-02-05 17:56:42 UTC

@netlify

netlify Bot commented Feb 5, 2026

Copy link
Copy Markdown

Deploy Preview for nx-dev ready!

Name Link
🔨 Latest commit 588c365
🔍 Latest deploy log https://app.netlify.com/projects/nx-dev/deploys/6984cc194382fd0008b5b28b
😎 Deploy Preview https://deploy-preview-34345--nx-dev.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@netlify

netlify Bot commented Feb 5, 2026

Copy link
Copy Markdown

Deploy Preview for nx-docs ready!

Name Link
🔨 Latest commit 588c365
🔍 Latest deploy log https://app.netlify.com/projects/nx-docs/deploys/6984cc191bad30000872344c
😎 Deploy Preview https://deploy-preview-34345--nx-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@FrozenPandaz

Copy link
Copy Markdown
Contributor

I'll handle this in #34329

@github-actions

Copy link
Copy Markdown
Contributor

This pull request has already been merged/closed. If you experience issues related to these changes, please open a new issue referencing this pull request.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Feb 12, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants