[teamleaderleo]

Leo, 2026-11-05: "honestly i'd rather us look at lot more at Petal and stuff like we ought to be able to get a lot more efficient and start going towards crazy stuff... after being able to do the basics end to end and adding eval measurements, ofc", and on the Petal list: "obv do all these and more".

HQ#7146 G5 (a person and an agent co-driving), asked for by the cu-codrive lane. Ports Petal's telepointers: cursors as metadata, never pixels (https://github.com/kiruna-labs/petal @ 7ac436a, Apache-2.0, src/telepointer.rs).

What changes

Today the WebRTC display stream draws the Mac's cursor into the video (showsCursor = true) and captures the Computer Use agent's cursor overlay window along with everything else.

• The viewer says it draws pointers. It sends a caps input with telepointers. An older host ignores it.
• The host leaves the cursor out and sends where the pointers are. For that viewer only, showsCursor turns off, applied with updateConfiguration without restarting the capture. The host then sends {t:"telepointers", person:{x,y}, agent:{x,y}}, each a 0–1 fraction of the streamed display and absent when off it. Messages go out on the unordered pointer channel (latest wins), sampled at 60 Hz, sent only when a pointer moved, plus once a second. Older viewers and the Mac viewer never send the capability, so they keep the cursor in the picture.
• The iPhone draws both (ScreenViewController). The person's pointer is the existing arrow, at the Mac's pointer position in direct mode; trackpad mode and a hovering iPad pointer keep their local cursor. The agent's pointer is a ring with a tinted centre. Both keep a readable size when zoomed out.
• Two app hooks, nil by default. The app wires them (cu-codrive's lane):
  • ScreenStreamHost.agentPointer: (@Sendable () -> CGPoint?)?: the agent's pointer in global display points (top-left origin, as CGEvent locations). The host converts it to the streamed display's fraction. Feed it from ComputerUseAgentCursorWindows.fraction(_:in:)'s tips (the overlay window centres).
  • ScreenCaptureBackend.excludedProcessIDs: (@Sendable () -> [pid_t])?: processes whose windows stay out of the capture (the Computer Use helper that owns the overlay windows). It is read at each capture start and every 2 s, and applied with updateContentFilter. It excludes by process, not window ID, so overlay windows created after the capture started stay out too.

This also helps the skip gate (#36065) and the bitrate: a pointer move no longer changes the picture, so it costs no encoded frame.

Numbers

Pending, from screen-stream-g0's harness (cmux#36043, not yet on feat-cmux-next). Its viewer doesn't send telepointers, so its rows show the default path unchanged. The expected wins are kbps while only the pointer moves, and pointer glass-to-glass: the marker is drawn from metadata, not decoded video. Measuring them needs a harness viewer that advertises the capability; I'll ask screen-stream-g0. Draft until then.

Tests

TelepointerTests (new): the capability check, display fractions on a second display, the message's encoding and old-viewer compatibility.

Observed / Not run

• Prepush gate: no new violations.
• Not run: the iOS build of CmuxScreenKitUI (no iOS compile route from this lane VM; CI's iOS simulator build covers it), a device session, the macOS package tests (CI).

Changelog

• Changed: the iPhone screen viewer draws the Mac's pointer and the agent's pointer itself instead of receiving them in the video.

🤖 Generated with Claude Code
Agent session: agent session

[teamleaderleo]

@azooz2003-bit, for your awareness. These are Petal ports on the WebRTC stream path (HQ#7146), all in CmuxVNCWebRTC and CmuxScreenKit, each its own draft until screen-stream-g0's harness gives before and after numbers:

• Screen stream (WebRTC): bitrate ceiling from bits per pixel, Petal's 0.18 #36062: Auto and Smooth Motion's bitrate ceiling from 0.18 bits per pixel per frame, clamped 4–16 Mbps; 120 fps keeps 24 Mbps.
• Screen stream (WebRTC): skip unchanged frames (Petal's dirty-rect gate), 1 s keep-alive #36065: skip complete SCK frames with no dirty rects, and a 1 s keep-alive in place of the 500 ms idle refresh.
• Screen stream (WebRTC): viewer demand, capture no larger than the iPhone shows (Petal) #36074: viewer demand. The iPhone reports how large it shows the stream; SCK captures at 960 to 3840 on the long edge with 6/3/30 s holds.
• iPhone screen stream: report the display's refresh rate (up to 120 fps) #36077: the iPhone reports its display rate (up to 120 fps).
• Screen stream (WebRTC): telepointers, the iPhone draws the person's and the agent's pointers (Petal) #36086 (this one): telepointers. A viewer that draws pointers gets the cursor left out of the video, plus telepointers messages; there are hooks for the agent's pointer and for leaving out the overlay's process.

Old viewers and the Mac viewer keep today's behaviour in each. None of them changes the cmux.rd/1 path. If one of them steps on something you have in flight, say so and I'll rebase around it.
Agent session: agent session

[teamleaderleo]

Review (codex gpt-6-astra high, router-review): BLOCK at c3eb9c1

BLOCK at c3eb9c1

Visual verdict: no images; Leo tries the build

1. [P1] [blocking] [category:reliability] [finding:F1] [outcome:pending] ScreenCaptureBackend.swift:245-263 can miss setDrawsCursor(false): if the capability arrives while capture() is awaiting startCapture(), the stream is stored with the old cursor-enabled configuration and no reconfiguration is triggered because only the frame rate is checked. The viewer then draws metadata over a video that still contains the cursor.

2. [P1] [follow-up] [category:correctness] [finding:F2] [outcome:pending] ScreenStream.swift:316-320 and ScreenCaptureBackend.swift:40-44 add the agent-pointer and exclusion hooks, but no code in this tree assigns either hook. The advertised agent marker therefore never appears, and overlay processes are never excluded.

3. [P2] [follow-up] [category:correctness] [finding:F3] [outcome:pending] ScreenViewController.swift:224-228 converts normalized pointers using the current frameSize and stores pixels. Metadata can arrive before the first video frame, producing (0,0); applyFrameSize does not recompute it, so the pointer can remain at the top-left until the next periodic resend.

Follow-ups:

• [P1] [follow-up] [category:correctness] [finding:F2] [outcome:pending] Wire agentPointer and excludedProcessIDs in the app host path.
• [P2] [follow-up] [category:correctness] [finding:F3] [outcome:pending] Store normalized coordinates or recompute pointer pixels whenever the frame size changes.

[teamleaderleo]

Replies to the router review at c3eb9c1. Fixed in e2d4fc7:

• F1: capture() now checks the rate and drawsCursor together once startCapture() returns. A setDrawsCursor(false) that arrives while the capture is starting now reconfigures the stream, so the cursor no longer stays in the picture.
• F3: the iPhone keeps pointers as fractions of the display and turns them into pixels when it draws (streamPixels). A pointer that arrives before the first frame, or before a size change, now lands in the right place.
• F2 (hooks not assigned): this is on purpose and split by owner. This PR adds the hooks ScreenStreamHost.agentPointer (global CG points) and ScreenCaptureBackend.excludedProcessIDs (the helper's pid), and cu-codrive connects them in the app from ComputerUseAgentCursorWindows. That was agreed with them before this PR. Until then the person's pointer still works as metadata, and the agent marker just doesn't appear, which is the same as today.

[teamleaderleo]

Review (codex gpt-6-astra high, router-review): BLOCK at e2d4fc7

BLOCK at e2d4fc7

Visual verdict: no images; Leo tries the build

1. • [P1] [blocking] [category:correctness] [finding:F1] [outcome:pending] ScreenCaptureBackend.capture applies excludedProcessIDs unconditionally at lines 238–240, before the viewer’s telepointers capability is known. A configured Computer Use helper is therefore excluded for legacy viewers that never advertise telepointers, even though those viewers still expect the cursor/overlay in the video. Make exclusion conditional on the negotiated capability and restore the normal filter for old viewers.
2. • [P1] [blocking] [category:reliability] [finding:F2] [outcome:pending] startTelepointers appends to session.tasks at line 566 without synchronization, while endSession iterates and cancels that same array at line 687. A capability message can race with disconnect/bye, causing concurrent Array mutation and possible crashes. Protect task-list access with the session lock or use a dedicated task property.
3. • [P2] [follow-up] [category:ux] [finding:F3] [outcome:pending] streamPerson and streamAgent are never cleared when a stream disconnects or reconnects. After a telepointer-capable host drops, the UI can draw stale markers over the next stream, including duplicating the cursor when the new host is legacy. Reset both values when replacing or disconnecting the viewer.

Follow-ups:

• [P2] [category:ux] [finding:F3] stale telepointer state across reconnects.

[teamleaderleo]

Review (codex, router-review): SHIP at 2d70492

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 2. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at 2d70492

Visual verdict: no images; Leo tries the build

• [P2] [follow-up] [category:correctness] [finding:F1] [outcome:pending] The production WebRTC host still constructs ScreenStreamHost with a bare ScreenCaptureBackend and never assigns either new hook (MacScreenRdHost.swift:185-186; hooks default to nil at ScreenStream.swift:325-333 and ScreenCaptureBackend.swift:36-45). Consequently the iPhone receives no agent pointer and the Computer Use overlay process is not excluded. Wire these providers into the app path or defer claiming both-pointer behavior.

• [P2] [follow-up] [category:reliability] [finding:F2] [outcome:pending] capture records excludedApplied = pids even when filter had to fall back to an unfiltered display because SCShareableContent failed (ScreenCaptureBackend.swift:239-241,274-280). The 2-second watcher then skips retries while the PID set is unchanged (294-295), so a transient enumeration failure can leave helper windows captured for the lifetime of that stream. Preserve an unapplied state or retry the filter.

Follow-ups:

• F1: Wire the pointer and exclusion hooks into the production host.
• F2: Retry exclusion after filter-enumeration failures.

[teamleaderleo]

Review (claude-opus-5-5): SHIP at 86a657d

Delta on top of Codex's SHIP at 2d70492 (read directly; one file): fixes Codex follow-up F2. filter() now says whether the pids were left out; capture() records excludedApplied only when they were, and the 2 s watcher skips updateContentFilter when the window list failed, so a transient SCShareableContent failure is retried instead of leaving the helper's windows captured. Compile check CmuxVNCWebRTC passed at this head (job 3cdbab5186ff88582a87e6f9). F1 (wiring agentPointer/excludedProcessIDs in MacScreenRdHost.serveWebRTC) is cu-codrive's app-side follow-up, sent to them with the exact site.