Skip to content

fix(cmux-config): macOS file watch uses kqueue, so the daemon sees acpmux bind its socket - #29685

Merged
lawrencecchen merged 5 commits into
feat-cmux-nextfrom
fix/base-red-acpmux-socket-hint
Nov 1, 2026
Merged

lawrencecchen merged 5 commits into
feat-cmux-nextfrom
fix/base-red-acpmux-socket-hint

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

On macOS an idle cmux-tui daemon never connected to acpmux when acpmux started after the daemon, as long as acpmux's folder already existed. cmux-tui::idle_wakeups cmux_next_daemon_idle_has_no_periodic_wakeups is red on macOS for this reason ("the daemon did not connect to acpmux within 3 s of its socket appearing", also when run alone, job 509e39756a834d8f1fc1b527).

The agent chat notices thread (agent_chat_notices.rs watch()) waits on cmux_config::watch for acpmux's socket. On macOS that watcher used FSEvents for every folder that holds a watched name. FSEvents does not report a Unix socket being bound at all. Measured with a raw notify FSEvents watch of the folder (scratch diag job 313c5cbdc973cc41db671b58, aws-m4pro-6, two tries): a socket bind gave no event within 8 s, both times; in the same folder a file create came 6 ms after the write and the socket's removal 1.4 ms after the unlink. The paths were already canonical (/private/tmp/...), so it is not a /var vs /private/var mismatch, and not lateness. The focused test below fails the same way through cmux_config::watch. FSEvents teardown is also slow on a loaded Mac: two existing tests, agent_chat_notices "shutdown/drop ends a watch that waits for acpmux", failed on the first try in 2 of 5 base jobs on macOS ("the watch outlived ...": the watch thread, which starts and stops an FSEvents stream, took over 5 s to end) and passed on the gate rerun.

macOS now has its own backend, cmux-config/src/watch/kqueue.rs, built from the kqueue code that already watched a missing folder's ancestors (ancestors.rs, removed):

  • every planned folder gets a vnode watch of its entries (NOTE_WRITE, plus NOTE_DELETE/RENAME/REVOKE for the folder itself);
  • each watched name that is a regular file gets a content and metadata watch (a write in place changes no entry);
  • a folder event names no entry, so each folder keeps the identity (dev, inode, mode, ctime) of its watched names and hints only when one changed; other entries of the folder never hint;
  • one thread blocks in kevent with no timeout outside a burst, so an idle daemon does not wake. Dropping the watcher triggers a user event; no fseventsd call on teardown.

The folders watched are the same as before (the settings file's folder, the nearest existing ancestor of a missing one, and acpmux's own socket folder); no new folder is watched. Linux (inotify.rs) is unchanged; portable.rs now serves only platforms other than Linux and macOS.

Testing

Red first: fefd9a968bf adds cmux-tui/crates/cmux-config/tests/watch.rs alone.

what ref job result
test watch_ (macOS, aws-m4pro-2) base + test fefd9a968bf 6a56674b79d3d0183e77ce84 watch_hints_when_a_socket_is_bound_in_a_watched_folder FAIL on the first try and on the gate rerun ("no hint within 2s of a socket bound at a watched name"); the file and other-entry tests PASS
notices shutdown/drop, test ends_a_watch_that_waits_for_acpmux, 5 jobs (macOS) base e820cedb59f 96c47517, df5a6aab, d24dd90c, 6d014553, f0872aaa failed on the first try in 2 of 5 base jobs (96c47517, df5a6aab: "the watch outlived ..."), passed on the gate rerun (gate-notice: FLAKY); the other 3 passed (0.5-5.9 s)
idle test, 5 jobs (macOS) head e22fa881f4e 1e8d3215, 4f1b24d2, 96510a02, b771dabb, 118fa038 5 of 5 PASS on the first try, no FLAKY (118fa038 also ran ..._ignores_writes_below_a_missing_settings_folder: PASS)
notices shutdown/drop, 4 jobs (macOS) head f4866419, 49c68bfc, 773bcf20, 2ff97975 4 of 4 PASS on the first try (0.15-4.3 s)
all watch (Linux: fmt, clippy -D warnings, tests) head 8af7def1c888a8747fdc0a82 pass; 108 tests passed, including the 3 new watch tests
all watch (macOS: fmt, clippy -D warnings, tests) head 21155f8bc4aacd339b653cd5 pass; 106 tests passed, including the 3 new watch tests and the 4 notices shutdown/drop tests, no FLAKY
all cmux_next_daemon_idle (macOS: fmt, clippy, tests) head 1b35d91ba116484703fbf888 pass; 2 of 2 idle tests passed
test cmux_next_daemon_idle (Linux) head 2a18462dda8e919c20df96fc 2 of 2 idle tests passed

The scratch diag ran on branch scratch/acpmux-hint-diag (never merged).

Observed / Not run

Observed: the jobs above, read from their logs (each attempt counted, gate-notice: FLAKY searched).

Not run:

  • b28f08d6a57ceaa0bde5dc08 (Linux) and d67189fe2e4f7f3e92e1b7f0 (macOS): the filter idle_wakeups is the test binary's name, and nextest filters test names, so 0 tests ran. They count as not run (d67189fe's fmt and clippy did pass on macOS).
  • Windows: portable.rs is now the backend only for platforms other than Linux and macOS, so Windows still uses notify's RecommendedWatcher (ReadDirectoryChangesW) with the same code as before; this PR only removed its macOS branches. #[cfg(target_os = "macos")] keeps kqueue.rs out of every other build. The PR's build-windows / build x86_64-pc-windows-gnu check compiles it (pass on this head); no Windows test of the watcher ran.
  • No tagged app build: the change is in the daemon's file watcher. The settings watcher's file behavior is covered by watch_hints_when_a_watched_file_is_created_written_and_removed and the idle tests above.

Performance

No perf numbers: this changes which kernel API wakes the watch thread, not a measured speed path. The idle wakeup check is cmux_next_daemon_idle_has_no_periodic_wakeups (and ..._ignores_writes_below_a_missing_settings_folder), which pass at the head on macOS.

Changelog

Fixed: On macOS, agent chat notifications start as soon as acpmux starts, instead of never when acpmux's folder already existed.

…(red on macOS)

The agent chat notices watch acpmux's socket with cmux_config::watch. On
macOS a folder that holds a watched name is watched by FSEvents only, and
binding a socket there gave no hint, so the idle daemon never saw acpmux
start (idle_wakeups.rs cmux_next_daemon_idle_has_no_periodic_wakeups).
Also pins the settings watcher's file cases and that other entries of the
folder do not hint.
…vents

The daemon's agent chat notices watch acpmux's socket with
cmux_config::watch. On macOS a folder that holds a watched name was
watched by FSEvents only, and binding a Unix socket in it gave no hint
(cmux-config tests/watch.rs red on macOS, job 6a56674b79d3d0183e77ce84),
so a daemon whose acpmux folder already existed never connected when
acpmux started: idle_wakeups cmux_next_daemon_idle_has_no_periodic_wakeups
"did not connect to acpmux within 3 s".

macOS now has its own backend (watch/kqueue.rs), built from the kqueue
code that already watched a missing folder's ancestors: a NOTE_WRITE /
delete / rename / revoke watch of every planned folder's entries, and a
content and metadata watch of each watched regular file (a write in
place changes no entry). A folder event names no entry, so each folder
keeps the identity (dev, inode, mode, ctime) of its watched names and
hints only when one changed; other entries of the folder never hint.
One thread blocks in kevent with no timeout outside a burst, so an idle
daemon does not wake. Linux (inotify.rs) is unchanged; portable.rs is
now only for platforms other than Linux and macOS.
@lawrencecchen lawrencecchen added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@cursor

cursor Bot commented Oct 31, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review e22fa88: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr29685 -f ref=fix/base-red-acpmux-socket-hint -f platform=macos -f backend_mode=local

Shipped in:

@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Review (claude-reviewer-kqueue): SHIP at e22fa88
reviewer-session: e3d79fcf-a3c9-42fb-b44f-9dbf9820350e
author-session: a2126edf238aabf9f
R9: pass | no silent skip: tests/watch.rs is #![cfg(unix)] only. All 3 tests ran on macOS (21155f8b, aws-m4pro-7) and on Linux (8af7def1). The idle tests are macOS-only, and the PR says so.
perf claim: none

Scope: cmux-config/src/watch.rs, watch/kqueue.rs (new), watch/portable.rs, watch/ancestors.rs (deleted), tests/watch.rs, the agent_chat_notices.rs comment and idle-wakeups.md. I read inotify.rs and the base ancestors.rs to compare.

What I checked:

  1. FFI on Darwin arm64 and x86_64. struct kevent is (uintptr_t, int16_t, uint16_t, uint32_t, intptr_t, void*) = 32 bytes, and KEvent matches it. timespec is (long, long), so i64 is correct on both arches. The constants are correct: EVFILT_VNODE -4, EVFILT_USER -10, EV_ADD 0x1, EV_CLEAR 0x20, NOTE_DELETE/WRITE/EXTEND/ATTRIB/RENAME/REVOKE 0x1/0x2/0x4/0x8/0x20/0x40, NOTE_TRIGGER 0x01000000, O_EVTONLY 0x8000, O_NONBLOCK 0x4. These are the same values as the shipped ancestors.rs, plus the file NOTE_* values.
  2. Change coverage. Create, socket bind, delete and rename-replace give a NOTE_WRITE on the folder. The name's identity (dev, ino, mode, ctime) then changes, so matters returns true. An in-place write or a chmod of a regular file fires the file's own watch. A folder that is removed or renamed gives GONE. Then rearm sees, through still_at, that the inode changed and moves the watch. A folder that appears on the way gets the ancestor NOTE_WRITE, then rearm. A change in the window between arm and snapshot is covered: rearm snapshots after EV_ADD, and the caller hints after every rearm. For a symlinked settings file, the file watch follows the link, and watch_paths also adds the target's folder.
  3. Loops. No hint makes the watcher wake itself again. Reads and opens fire no NOTE_ATTRIB. This is proven because the idle test has the settings file in place, the first hint reloads (reads) it, and the test passes 4 of 4 on macOS. A snapshot is updated each time it is compared, so a stale ctime gives at most one extra hint, not a loop.
  4. Descriptor or ident reuse. One thread owns the kqueue and all watch descriptors. A close drops its knote and the queued events, so a reused fd number cannot get a stale event. The kqueue is not inherited by fork, and the opened files have O_CLOEXEC.
  5. Idle wakes. Outside a burst, kevent has no timeout, so no periodic wake is possible. Busy folders (an ancestor such as $HOME, /Library, or ~/.acpmux while acpmux runs) wake the thread only when their own entries change. The cost of each wake is one stat per watched name, and no hint is sent. This is the same as the base for ancestors (ancestors.rs). It is narrower than the base FSEvents behaviour for the folders that hold a watched file.
  6. Shutdown. Drop sends an EVFILT_USER NOTE_TRIGGER and joins. It does not join when it runs on its own thread. The kqueue Arc keeps the fd alive until the thread exits. This removes the fseventsd teardown delay that made the notices drop/shutdown tests fail 2 of 5 at base.
  7. Tests. The socket test failed twice at the red commit (6a56674b) and passes at the head. The file and other-entry tests guard parity. The timing margins are generous (2 s hint window, 800 ms no-hint window, a scratch folder per test and per pid), so I expect no flakiness.
  8. Docs: accurate. No stale references to ancestors.rs or cmux-config-ancestors remain in the repo.

Evidence I read: the head macOS all watch nextest summary in 21155f8bc4aacd339b653cd5 (106 passed, including the 3 cmux-config::watch tests and the agent_chat_notices drop/shutdown tests; the job was still in its later steps when I checked), and the job list in the PR description. I did not read the logs of the other jobs.

Residual risks (none block):

  • A folder the user cannot read (EACCES on open) cannot be watched. When it is the only folder in the plan, the watch fails to start. Otherwise, after a rearm that skips it, nothing watches it. Base FSEvents did not need read access, and Linux inotify has the same limit. /Library/Managed Preferences/ is normally 755, so this is unlikely.
  • A watched folder whose parent is renamed keeps its watch on the moved inode. Linux has the same behaviour.
  • On macOS a chmod of a socket or of a missing-folder name does not hint, because the folder watch has no NOTE_ATTRIB. No current user of the watcher needs this.
  • The watch.rs module doc says "a close after a write". On macOS each write hints, and the burst coalesces the writes. This is minor.

@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Checks at head e22fa88: 68 pass, 77 skipped, 2 pending (R2 publish, Sentry dSYM upload). One non-green check:

  • pr-red-checks / comment: infrastructure. Its job ended cancelled in run 41242388813 (a status-comment job, no test or lint step failed). Rerun on the same SHA.

The cmux-tui Rust tests for this change ran on the fleet (see Testing in the description): macOS fmt, clippy and tests in 21155f8bc4aacd339b653cd5 and 1b35d91ba116484703fbf888; Linux in 8af7def1c888a8747fdc0a82 and 2a18462dda8e919c20df96fc.

@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Review (claude-reviewer-kqueue; carried): SHIP at beb78b1
reviewer-session: e3d79fcf-a3c9-42fb-b44f-9dbf9820350e
author-session: a2126edf238aabf9f
R9: pass | no silent skip: tests/watch.rs is #![cfg(unix)] only. The 3 watch tests ran on macOS (21155f8b) and Linux (8af7def1). The idle tests are macOS-only, and the PR says so.
perf claim: none

This carries my SHIP at e22fa88 (#29685 (comment)) to the merge head beb78b1. I did a read-only check:

  1. beb78b1 is a merge. Its parents are e22fa88 (the reviewed head) and aaa0170 (feat-cmux-next). The merge-base is e820ced, the base of the review.
  2. The PR patch is the same. git diff e820cedb59fc e22fa881f4e | git patch-id --stable and git diff beb78b1b87a1^2 beb78b1b87a1 | git patch-id --stable both give 0f1edb75256a7f7ca2c30c2cfb48bc51b9f86944.
  3. Nothing relevant moved in the 41 feat commits (e820ced..aaa0170). No commit touches cmux-tui/crates/cmux-config, agent_chat_notices.rs, mux/settings.rs, mux/keybindings.rs, tests/idle_wakeups.rs or plans/cmux-next/idle-wakeups.md. At the new head, the users of cmux_config::watch are the same as before: settings.rs, keybindings.rs and agent_chat_notices.rs. Cargo.lock changed only by adding the cmux-vm crates and their dependencies (da664a2). The notify version did not change.
  4. New evidence at e22fa88 that I read in the job logs:
    • macOS all watch 21155f8bc4aacd339b653cd5 is done: 106 tests run, 106 passed.
    • macOS idle 1b35d91ba116484703fbf888: both idle tests PASS (nextest 2/2). The job state still showed running.
    • macOS idle 118fa0381c4a89f6a20e4850 is done: 2/2 PASS.
    • I could not read the Linux idle job 2a18462dda8e919c20df96fc: its controller log shows no test lines. I did not rely on it.

The residual risks in my first receipt are unchanged.

@lawrencecchen lawrencecchen added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 31, 2026
@cursor

cursor Bot commented Oct 31, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 1, 2026 00:49 Inactive
@blacksmith-sh

blacksmith-sh Bot commented Nov 1, 2026

Copy link
Copy Markdown

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Red check classification on head 0558e13 (none is this PR's own red):

@lawrencecchen
lawrencecchen merged commit 521b0bb into feat-cmux-next Nov 1, 2026
217 of 223 checks passed
@lawrencecchen
lawrencecchen deleted the fix/base-red-acpmux-socket-hint branch November 1, 2026 02:20
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
Second review of #30028: since #29685 cmux_config watches every directory
with kqueue on macOS, not FSEvents on the parent. Comment only.
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
…nd made again (#30028)

* test(cmux-tui): a customize folder deleted and made again reloads a later save

Red first: on Linux the inotify watch goes away with the old folder, and
nothing watches the new one until the daemon restarts.

* fix(cmux-tui): watch a local apps folder again after it is deleted and made again

On Linux the inotify watch on the customize folder ended with the old
folder (or stayed on it after a rename), so a folder replaced by a git
checkout, an editor or rm -rf and a copy was not watched until the daemon
restarted, and its saves did not reload.

The folder's entry in its parent is now watched through
cmux_config::watch (the same Plan the settings watch uses: it climbs to
the nearest existing ancestor while the parent is missing too). On each of
its hints a folder whose device and inode differ from the watched one is
watched again, before the read. No timer and no polling. The local
development folder gets the same treatment.

macOS: FSEvents watches a path, so a stream there kept reporting the new
folder; the same code re-arms it once (one stream restart, then a read).
The missing-parent case (the parent of the folder also missing at start)
was unwatched on both platforms before and is now covered.

* test(cmux-tui): a customize folder whose parent is missing at start installs once made

The watch went no higher than the folder's parent, so with ~/.config/cmux
missing at daemon start nothing picked the folder up until a restart.

* fix(cmux-tui): arm the local apps folders again on every folder hint

75c3cff compared the folder's device and inode to decide whether to watch
it again. A folder deleted and made again at once often gets the freed
inode number back (ext4), so the new folder looked like the watched one
and stayed unwatched: the replace test failed on Linux in job
3baeff7475c95b47351b7de9 and passed in 1579c030db74662cf3887cb1 with
other tests beside it.

Each folder hint now drops the folder's old watch and watches what is at
the path. notify's own event for the watched folder itself (removed or
renamed) also arms again, because notify ends its record of a deleted
folder's watch only when it reads that event, and read late it can end
the watch just armed. The entry watch starts before the folders are armed
and its first hint (armed) is skipped, so startup arms and reads once.

A unit test drives a real notify watcher: a folder deleted and made
again, then renamed away and made again, is watched once armed again,
and the old folder's watch names nothing under the path.

* fix(cmux-tui): drop the notify::Watcher import local_reload.rs no longer uses

Folders::arm takes the watcher as impl notify::Watcher, so the trait
import is unused and cmux-tui-core fails clippy -D warnings.

* docs(cmux-tui): say what the local apps re-arm costs on macOS

Review of #30028: the macOS entry watch is FSEvents on the parent or kqueue
on an ancestor, and arming again restarts the FSEvents stream twice per
folder (unwatch, then watch). Comment only.

* docs(cmux-tui): the macOS entry watch is cmux_config's kqueue watch

Second review of #30028: since #29685 cmux_config watches every directory
with kqueue on macOS, not FSEvents on the parent. Comment only.

This branch was previously deployed

3 inactive deployments
ci-routing-agent — 0558e130 Deployed Nov 1, 2026 by github-actions[bot] via cmux-next macOS placement #56573
artifacts — 0558e130 Deployed Nov 1, 2026 by cmux-bot[bot] via add the Windows daemon to the tree #92381
sentry-dsyms — 0558e130 Deployed Nov 1, 2026 by cmux-bot[bot] via upload cmux-tui dSYMs to Sentry #92381
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant