Repository navigation
cmux-next: Mac Screen tabs and the device hub (per-device access, no pairing links) - #22788
Conversation
…reat a close during auth as an auth failure
RemoteInputController holds its sink weakly and nothing else held the sink handed to RemoteDesktopPane, so control-mode input went nowhere. Red on the base (Expectation failed: pane.view.capture.controller.sink != nil).
- CmuxNextRemoteView: decoded-frame sources (RemoteViewFrameSource), fit scaling for sources that keep their own screen size, the VNC compatibility path badge, a sign-in end state, and via=vnc&port= tab records. VNC hosts may be on loopback, the tailnet or a private LAN (RFB streams are not encrypted); cmux.rd/1 stays loopback-only. - RemoteDesktopPane keeps its input sink alive (the controller holds it weakly, so control-mode input was dropped). - CmuxNextApp: RemoteVNCSource/RemoteVNCInputSink over CmuxVNC, Keychain sign-ins, a sign-in sheet, and host=mock&via=vnc as a loopback VNC demo. - CmuxVNC CI workflow; iOS demo app experiments/cmux-vnc-ios (cmux VNC).
…OS run scheme, CPU snapshots, dark session chrome
… 26.3 Checked by hand against macOS Screen Sharing on aws-m4pro-1 (team cmuxvnc account): RFB 3.8 via 003.889, Apple DH (type 30), ZRLE, 2944x1080 frame.
- CmuxVNC: cap framebuffer, cursor and Apple DH sizes; reject rectangles outside the framebuffer; decode Raw row by row; bound zlib output by the rectangle; compare the 3.3 security word whole; never answer type 2 with an account password; honor 0xFFFF only with LastRect; Latin-1 clipboard; an open that is cancelled or closed before the connection starts fails instead of hanging. - Policy: dotless numeric names (134744072, 0x08080808) are public IPv4. - App: local/localhost never resolve; a tab closed while the demo server starts stops it; sign-ins are saved only after the host accepts them; late events from an older session no longer overwrite the status. - Remove a stray copy of the package committed by mistake.
… (SSH cannot unlock the Keychain); never echo them
…ain, file deleted)
Tap clicks where you touch, press-and-hold then drag drags, one finger pans a zoomed view, two-finger tap right-clicks, two fingers scroll what is under them, pinch zooms.
- CmuxVNC: ExtendedDesktopSize screens event; VNCScreenLayout finds the displays in a combined framebuffer (macOS Screen Sharing sends one image for all displays and no layout): runs of columns with the same black void, an unpainted display kept full height, equal split for very wide frames. Checked on aws-m4pro-1: 1920x1080 + 1024x768. - cmux VNC iOS: each display is a page; swipe or tap the dots to switch; pinch leaves paging. Opens on the main display.
Fitted mode pins the view to one display: zoom out stops at its fit, panning stays inside it, and a swipe at fit pages to the next display. Orientation: Auto-Rotate, Portrait or Landscape, remembered.
…host, signaling Worker - experiments/cmux-vnc-ios/signal: cmux-vnc-signal Worker + Room Durable Object (deployed): host/viewer WebSocket relay gated by a room secret (only its SHA-256 is stored), ICE servers (Cloudflare STUN; Realtime TURN when CLOUDFLARE_TURN_KEY_ID/_API_TOKEN are set). - Packages/Shared/CmuxVNCWebRTC on libwebrtc M154 (the CmuxLink pin): VNCPairing links, VNCPeerLink (ordered reliable 'rfb' channel, 16 KB chunks, buffered-amount backpressure), WebRTCRFBTransport for VNCClient, VNCWebRTCHost bridging the channel to a VNC server over TCP, and the cmux-vnc-host CLI (persistent 0600 pairing, terminal QR). - Live test through the deployed Worker: connected in 0.34 s, 20 frames.
… Mac shortcuts bar - Pair a Mac running cmux-vnc-host from its QR code or cmux-vnc:// link; paired Macs connect over WebRTC through the cmux-vnc-signal Worker. - One-tap Mac shortcuts under the toolbar: Spotlight, Mission Control, App Windows, Switch App, Copy, Paste, Undo, Select All, New Tab, Close, Quit, Screenshot, Force Quit, Lock Screen. - The stats line names the path (WebRTC, Direct, Loopback).
…), trim secrets, report TURN status TURN (Cloudflare Realtime key cmux-next-ios-wrtc, from 1Password) is set on the Worker in Lawrence's Cloudflare account; /ice now returns STUN, TURN and TURNS servers.
…update coverage stat
- Host logs the ICE path, the viewer's candidate classes, and every 2 s: RTT, server→viewer rate, data channel backlog. - iOS perf.log: per update (server latency, receive+decode, bytes, pixels, main-queue wait, copy, draw) and per input (input → next frame); -perfScript plays Spotlight/type/Escape/click/scroll rounds on a saved Mac. - Throughput test (two peers forced through the relay, 20 MB): TURN UDP 1.8 MB/s or no path, TCP 3.1 MB/s, TLS 5.4-5.9 MB/s; direct 7.6-9.3 MB/s. Relays now drop UDP TURN when a TCP/TLS relay is offered. - Phone before this fix (relay over UDP): input → frame 3.2-3.5 s, relay throughput 0.45-0.56 MB/s, up to 2 MB backlog; phone copy+draw 4 ms.
…er a host reset); re-pairing replaces the old entry A host whose room socket reset left a dead socket in the Room; the next viewer join threw 'Can't call WebSocket send() after close()' (1101) and the phone never reached the host. Sends now tolerate dead sockets. A running Durable Object keeps old code until it restarts, so the host was re-paired (new room). LinkViewerTests connects to a running host from a Mac.
…, input on a data channel - ScreenStreamHost (cmux-vnc-host --stream): frames from this Mac's own Screen Sharing over loopback (no Screen Recording grant needed), one display cropped per viewer (RTCCVPixelBuffer crop, no copy), screencast video source, maintain-resolution, idle refresh; JSON input (move, scroll, key, chord, text, display) back to Screen Sharing. - ScreenStreamViewer: receive-only video transceiver + input channel, keeps the track (libwebrtc drops renderers with the wrapper), live stats. - H.264 first and offered at level 5.2: at level 3.1 a frame above ~720p fails encoder setup and libwebrtc silently falls back to software VP8. - Pairing links carry kind=stream. - Live test through the Worker: hardware H.264 both ends (VideoToolbox), first frame ~2 s, 5 s of the animated 1280x800 demo in 89 KB, typing reaches the server.
… streams the cmux account The iOS app plays a kind=stream pairing as a WebRTC video track (ScreenVideoView over RTCMTLVideoView), sends input on the data channel, pages displays by asking the Mac, and shows WebRTC's stats (fps, KB/s, RTT, codec, hardware decode, direct/relay). Phone measurement through the TURN relay: input → next frame p50 84 ms, p90 322 ms (was 0.8-3.3 s with RFB bytes over a data channel).
… unordered pointer channel - ScreenBackend seam: VNCLoopbackBackend (V1) and ScreenCaptureBackend (ScreenCaptureKit, native pixels, 60 fps, NV12 straight to the hardware encoder; CGEvent input with click counting, wheel, key codes, Unicode text). cmux-vnc-host --capture; make-host-app.sh wraps it as CmuxScreenHost.app (WebRTC.framework embedded) for its TCC grants. - WebRTC used as designed: no SDP edits (H.264 level 5.2 declared by a decoder factory, codec order via setCodecPreferences:error:), video lets ICE choose any relay transport, unordered 'pointer' data channel for pointer moves (latest wins), ordered 'input' for buttons and keys. - Simulator baseline on mini 2 (V1 source): input → first frame p50 68 ms, p90 610 ms; the tail is WebRTC's CPU-overuse adaptation (BGRA→NV12 on the CPU), which ScreenCaptureKit's NV12 frames remove.
…GPU NV12 - experiments/cmux-vnc-ios/host/marker: a counter window (8 B/W cells) that increments per key and pointer move; MarkerLatencyTests reads it from decoded WebRTC frames and times every input in bursts (keys at 20 and 50 Hz, moves at 60 Hz, keys at 2 Hz) with a first/last-quarter trend. - V1 backend: one ordered input queue instead of a task per input. Before: 50 Hz keys p50 2.9 s with 20/60 never seen, 60 Hz moves 108/120 never seen. After: all inputs seen; 50 Hz keys p50 205 ms, 60 Hz moves 282 ms. The remaining stalls (2 fps, up to ~2 s) are the Screen Sharing frame source; V2's ScreenCaptureKit source replaces it. - BGRA→NV12 on the GPU (VTPixelTransferSession) before encoding. - -perfScript also runs from -autoLink (simulator runs).
…presets A room (signaling) connection dropping no longer ends a running stream: once WebRTC is connected only WebRTC ends the session. The viewer reopens its room socket quietly and offers once per session; the host ignores a viewer's room drop for an open session. Room sockets send keepalive pings. The iOS app reconnects by itself when the stream drops (backing off, up to five tries) and right away on returning to the foreground. Quality menu: Auto, Smooth Motion, Fixed Low/Medium/High, applied with WebRTC's sender parameters and kept per Mac. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new Rust workspace that speaks the Swift host's wire protocol, so the cmux VNC iOS app pairs with it unchanged: - cmux-screen-proto: pairing links, room signaling, input and host messages, quality presets (tests pin the Swift formats). - cmux-screen-host: room client with keepalive, WebRTC sessions on libwebrtc (LiveKit's crate), quality via sender parameters, room drops never end a connected session; macOS backend with ScreenCaptureKit zero-copy NV12 to VideoToolbox and CGEvent input; libwebrtc's desktop capturer as the shared Windows/Linux capture path (input there is next). Measured on mini 2: consecutive-input glass-to-glass p50 40-63 ms (Swift V2 51-55 ms); iOS simulator input to first frame p50 30 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…; log capture rate A display larger than the viewer's H.264 level (5.2) takes is captured by ScreenCaptureKit scaled to fit 3840x2160 (aspect kept, even sizes). The capture thread logs complete and idle frames every 3 s next to the encoder line, which showed the encoder keeping up at 56-57 fps on a busy screen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Sessions no longer leak: channel handlers capture the label, not the channel, and are cleared when a session ends (they held the backend and its last frame). Measured: host memory flat at 52 MB over 4 sessions. - The ICE server request times out after 10 s instead of holding the host. - Fixed quality presets set the connection's bitrate floor (encodings have no minimum in this API); adaptive presets restore libwebrtc's 30 kbps. - ScreenCaptureKit start, display switch and stop run off the host loop. - pairing.json is created 0600 in a 0700 folder (no readable window). - A previous session's late ICE candidates are not sent to the next viewer. - Text is typed one grapheme at a time; captures are tagged sRGB; a literal + in a pairing link stays a +; a short room id cannot panic the log line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Same sources as cmux VNC with its own bundle ID (dev.cmux.vnc.rust), display name and rust-colored icon, so it installs next to the Swift-host app. It registers no cmux-vnc:// scheme, so pairing links keep opening cmux VNC. The home title shows the bundle's display name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Moves the iOS demo's session and screen code into Packages/Shared/CmuxScreenKit: - CmuxScreenKit (iOS, macOS): ScreenSession (stream over WebRTC, RFB over WebRTC, direct VNC, demo desktop; reconnects with a configurable limit, quality, input, stats), ScreenPreferences, OrientationLock, an opt-in ScreenPerfLog; re-exports CmuxVNC and CmuxVNCWebRTC. - CmuxScreenKitUI (iOS): ScreenSessionView with ScreenFeatures (shortcuts bar, quality menu, orientation lock, fit toggle, display paging, stats, keyboard, send-keys menu, refresh, hide toolbar, resume on foreground), ScreenView / ScreenViewController (fitted mode, paging, the touch input model, keyboard), MacShortcutsBar (custom rows), ScreenOrientation, PairingScanner. Both demo apps now build on the library (no screen code of their own). README documents adding it and each feature. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fix: the prompt kept its fonts when only the type scale changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review (subagent, correctness first): SHIP at 0491de9 Scope: the AppKit port of Mac Screen's views (ffc998b..01a89d6), the visual parity and review repair e5056c4, the approval prompt fix 0491de9, and the merges of Pass 1, the port (ffc998b..01a89d6): FIX, with one medium finding: text didn't follow density, font size or interface scale changes.
Pass 2, the repair and the merges (e5056c4, 5799670, 14f0833): FIX, with one medium finding: the approval prompt updated
Pass 3, 0491de9: SHIP. The commit is exactly the suggested one-line condition. It rebuilds and resizes an open prompt and introduces no loop. Fleet at e5056c4:
Fleet at 0491de9:
Visual parity: the AppKit port and the SwiftUI build were compared on one mini, light and dark. The home page, the device list, the tab toolbar, the connect and end cards and the approval panel match after e5056c4. Details are in the PR body's "AppKit port" section. |
… the owned pool Reds this PR caused on 0491de9: - crash-class rules: the AppKit port's super calls (NSButton performKeyEquivalent, NSControl keyDown/layout, NSTextField layout) were not in the selector table; probed with check-objc-selectors.py --probe --write on macOS 27.0.1 (all answered). - defaults-snapshot / check-menu-actions: the port's one menu row builder (NSMenuItem.macScreen, the toolbar's Display, Quality and Send Keys menus and the device row's More menu) replaced SwiftUI Menu buttons the check could not see; baselined with the reason, as earlier ports were. - CI guard group: the new cmux-vnc.yml read vars.CI_PR_POOL_OWNED, which owned_pool_rescue.py does not watch; it now runs on the paid runner lanes only (tests/test_ci_owned_pool_sources.py passes). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review (subagent, correctness first): SHIP at 28ba4aa Scope: 28ba4aa on top of the reviewed 0491de9. It fixes the three reds this PR caused at 0491de9.
The remaining reds at 0491de9 are base reds, also red at base tip 95b7af2 or on unrelated PRs:
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
|
Review (subagent, correctness first): SHIP at 5657e60 Scope: the merge of
|
Conflict: the + menu's ranks (c0c88e7 lists tab kinds first). New Mac Screen Tab is a tab kind: rank 3, after New Agent Chat and before the profile and New Tab Page rows; ContextMenuStandardTests lists it there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
|
Review (subagent, correctness first): SHIP at afb80c2 Scope: the merge of
|
The base's B1 refactor (ba5e6df) moved RemoteRdDebugSource.swift into CmuxNextRemoteView with package access; this branch had moved it unchanged into CmuxNextRemoteViewSessions. Keep the base's copy: RemoteViewSessions depends on CmuxNextRemoteView, so RemoteViewPageSession sees it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new sleep-poll guardrail (base ceebde5) flags the ping loop. A keepalive is due on a fixed interval by protocol and waits for no condition, so there is no event to await instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review (subagent, correctness first): SHIP at 3a80b42 Scope: the merge of The merge
3a80b42. The new base guardrail On reds: |
The only conflict was the settings source stamp; regenerated with defaults_guard.py source-stamp (check-stamp and check-fresh pass). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review (subagent, correctness first): SHIP at ada5350 Scope: the merge of
|
SettingsSchema.swift: keep the base's HangReportsSettingsSchema and this branch's MacScreen, MacScreenDebug and DeviceHub descriptors. The generated exports (docs/mdm/*, settings-schema.json) take the base side for the autofix to regenerate; the source stamp is regenerated with defaults_guard.py source-stamp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…leton-allow The base's new singleton guardrail reads the moved line as new (its access changed for the move out of CmuxNextApp). It is a DEBUG-only weak registry the debug.remote_view socket method reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review (subagent, correctness first): SHIP at 16a107e Scope: the merge of The merge
16a107e. The base's new Checks: |
Conflicts: schemas/defaults/cmux-next.json (union of base hangReports and the PR's deviceHub/macScreen defaults) and the settings export stamp (regenerated with defaults_guard.py source-stamp). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review (subagent, correctness first): SHIP at c3c9521 Scope: the merge of The merge
Checks: fleet compile-check 40004702940 succeeded. |
Only conflict: settings source stamp, regenerated with defaults_guard.py source-stamp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CmuxNextDevicesTests uses daemonSwiftSettings, which set no MainActor default isolation, so Locked<Value>'s implicit deinit is nonisolated and the swift-frontend -O crash the new base guardrail guards against does not apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review (subagent, correctness first): SHIP at 37c71c0 Base merge of feat-cmux-next 15b43c2 into c3c9521 (previously reviewed head) plus one escape-comment commit. Fresh review subagent, read-only:
Findings: none. |
Brings in #21233. Only conflict: the generated settings source stamp, regenerated with scripts/ci/defaults_guard.py source-stamp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review (subagent, correctness first): SHIP at 79ad552 Merge of base 0a2a8e4 (brings in #21233) into 37c71c0. Fresh review subagent findings:
|
|
Red classification at 79ad552 (none touch this PR's files):
|
|
Merged by hand at the user's direction while next-autoland has posted no verdict since 12:21Z. Exact head 79ad552 has a SHIP review receipt and a green fleet compile-check (run 40024719066); remaining reds are base or infra, classified above. |
cmux NEXT gets a Mac Screen tab: see and control another Mac signed in to the same account, without pairing links. Each Mac registers a device key with a device hub, the host decides per device who may view or control it, and every session starts from a single-use ticket the hub mints. Settings › Devices lists your Macs, their permissions, relays and history; a menu bar item on the host shows who is connected and stops or revokes them.
This replaces #20426 (Mac Screen on
feat-cmux-screen-kit) and the stacked device-auth branch; both are folded in here and merged with currentfeat-cmux-next. The active-team picker (#21233) lands separately.What it adds
CmuxNextAppMac Screen pane onCmuxScreenKit/CmuxVNC): live H.264 stream, fit/letterbox, toolbar with transport, fps, RTT and codec, Quality, Send Keys, Refresh, Disconnect. Input goes to the host only while the tab's window is key.workers/device-hub, new): a Cloudflare Worker with a per-user SQLite Durable Object (devices, permissions, relays, audit) and a per-session Durable Object. Clients hold one hibernating WebSocket each, so permission and device changes arrive in real time; nothing polls. Contract:docs/device-hub/PROTOCOL.md.CmuxNextDevices(new package target): device identity, hub client, access state,RelaySecretVault(an actor; Keychain work never runs on the main actor).cmux-vnc://pair?…links, saved link Macs and the confirm gate are gone. Pairing-link text typed into New Tab or the omnibox is refused, never searched, sent to an agent or loaded.Security model
RelaySecretVault, off the main thread. A Mac added later gets a relay's secret once someone saves the relay again.Backend
cmux-device-hub-dev.debussy.workers.dev(deployed, secrets set).cmux-device-hub.debussy.workers.dev(deployed). Its TURN secrets aren't set yet, so production relays through the cmux TURN service won't work until they are; direct and user-relay paths don't need them.workers/device-hubis under the Business Source License 1.1 (workers/device-hub/LICENSE).AppKit port
feat-cmux-nextnow forbids new SwiftUI in macOS packages (scripts/cmux-next/check-no-swiftui.py, plans/cmux-next/no-swiftui-DESIGN.md), so the Mac Screen views are AppKit:The new files are
MacScreenButtons,Controls,HomeParts,ToolbarandCardView.check-no-swiftui.pyreports 0 Mac Screen findings; the 17 findings it still prints are staleCmuxNextAppsallowlist lines on the base. The port keeps every localization key, accessibility identifier and action.Review: a correctness-first subagent reviewed the port and found one medium issue. Text didn't follow density, font size or interface scale changes. It is fixed in e5056c4: the home page, cards and approval panel rebuild when the type scale changes, and the toolbar resets its fonts. Low findings left for follow-up:
Visual parity: I compared the port with the SwiftUI build (combo3) on one mini, at the same window size, with the same account, in light and dark. The screens compared are:
The comparison found these differences, fixed in e5056c4:
Remaining differences:
Fleet runs at e5056c4: suites
MacScreen,CmuxNextDevicesTests,DeviceHubandSettingsall pass, andcheck-release-compile.shpasses.Verification
Fleet (
cmux-ci run,scripts/ci/package-test-lane.shonPackages/macOS/CmuxNext), each fix landed red first:CmuxNextDevicesTests,MacScreen,NewTabIntentTableTests,SearchEngineEveryPathTestsgreen at 372113a (h7fix-after-372113a7) and on the team-picker combination 863d515 (h7fix-try-863d5155).DeviceHubSettingsTests,CmuxNextDevicesTests,DeviceHub,MacScreengreen at c1850d1;CmuxNextSettingsTests,DeviceHubSettingsTestsgreen at ece4553.CmuxNextDevicesTests,MacScreen,SettingsPage,CmuxNextSettingsTestsgreen at b649b93.workers/device-hub: 36 vitest tests (workers pool).These ran before the merge of current
feat-cmux-next(3265798) in 3562ddd; this PR's CI covers the merged head.Live walkthroughs on two real Macs (agents drove the UI; agent verdicts, not a person's):
DeviceKeychain) are fixed in 372113a with red-first tests above.DeviceKeychainstalls; the longest viewer stalls were 455 ms outside the device paths. Not reached: most of the access matrix, all 20 revocation cells, Relay Only through a user relay, team switch during a session, reconnect, and the H7 fixes in the live UI (Ask for Access, refresh after reconnect, link refusal, removal cleanup), which are covered only by tests.Review fixes after these runs (see the review comment): no ticket for a session that ends mid-approval, identity Keychain work off the main thread, lowering to Ask ends live sessions. Worker 74/74 and fleet
DeviceHubIdentityThreadTests,CmuxNextDevicesTests,MacScreen,DeviceHubSettingsTestsgreen on the fixed code (the fleet run needed the one-line base compile fix from #22821).Known open item: the user reports pointer movement on a Mac Screen session feels slower than earlier builds. The cause hasn't been measured.
No capture: a Mac Screen session needs two signed-in Macs on the device hub, which a single fleet capture mini cannot provide; the two-Mac walkthroughs above are the UI evidence.
Since d81c180: merged current
feat-cmux-next(dfd90dd, d3b87f7); the Mac Screen hosting view is no longer generic, which fixes the Release optimizer crash on its deinit (fleetcheck-release-compile.shgreen at 588e85e, step 2837c2b8); the connected dot uses the lit glyph color (base lint rule); the New Tab omnibox action handles the pairing-link refusal; the cmux-screen Cargo workspace is listed as skipped in.github/rust-workspaces.toml; settings exports restamped from the fleet. The Settings page'swriter()no longer requirescontext.userGesture(7c98ac8, so accessibility presses count as the user): a trust-policy change, disclosed here.Changelog
🤖 Generated with Claude Code