Repository navigation
cmux mcp: video_frames and video_info, so an agent looks into a video without ffmpeg - #30921
Conversation
… without ffmpeg Leo, 2026-11-01: "i feel like cmux-next should give u better tools to deal with this kind of thing". The server picks times, sizes, the byte budget and scene changes (Rust); the app decodes through AVFoundation (video.frames). Agent-Session: https://cmux.com/deeplink/agent?id=aid-bbe6e8ca1bc11f92f25b Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub Actions
Inactive
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
red: cmux-next checks (god files, concurrency, crash safety, l10n); NEW: Default changesThis pull request changes 9 defaults at
Needs a review. It passes with an independent review receipt on this commit: a comment or review whose first line is Merge receipt: merged as Regenerate the snapshot with Shipped in:
|
|
Owned by an agent session · session |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
9 issues found across 8 files
You’re at about 94% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="cmux-tui/crates/cmux-tui/src/cli/mcp/video_tools.rs">
<violation number="1" location="cmux-tui/crates/cmux-tui/src/cli/mcp/video_tools.rs:320">
P3: `scenes: true, count: 1` always reports that the picture barely changes, even when scene samples contain large changes. Remove that claim or pass whether any change was detected separately from the selected frame count.</violation>
<violation number="2" location="cmux-tui/crates/cmux-tui/src/cli/mcp/video_tools.rs:370">
P3: When the file is missing on this machine, the error only says 'cannot read ...'. The module doc says remote chats are told the file is not on this Mac and to use that machine's tools, so this message should say the file is not on this machine and point to that machine's tools.</violation>
</file>
<file name="Packages/macOS/CmuxNext/Tests/CmuxNextAgentPaneTests/AgentVideoFramesTests.swift">
<violation number="1" location="Packages/macOS/CmuxNext/Tests/CmuxNextAgentPaneTests/AgentVideoFramesTests.swift:28">
P2: A writer failure can make this loop spin forever: the helper records failures with `#expect` but never aborts or checks `writer.status`. Use throwing requirements for writer startup/appends and bound or status-check the readiness wait so codec or disk failures fail the test rather than hanging CI.</violation>
<violation number="2" location="Packages/macOS/CmuxNext/Tests/CmuxNextAgentPaneTests/AgentVideoFramesTests.swift:88">
P2: Guard the returned frame counts with `try #require` before subscripting; otherwise a decode failure crashes the test process instead of reporting which expectation failed. Apply the same protection to the `frames`, `samples`, and `one.frames` subscripts in this suite.</violation>
</file>
<file name="cmux-tui/crates/cmux-tui/src/cli/mcp/code_mode.rs">
<violation number="1" location="cmux-tui/crates/cmux-tui/src/cli/mcp/code_mode.rs:994">
P2: In one script, a second `video_frames` call (or a call after a computer use screenshot) can exceed the shared 8 MiB image budget. Its frames then come back as `dropped: true` markers, and the call's text still says the frames are 'below'. Surface the script-level drop in the text, or count the video budget against the shared image budget before answering.</violation>
</file>
<file name="Packages/macOS/CmuxNext/Sources/CmuxNextApp/Control/VideoControl.swift">
<violation number="1" location="Packages/macOS/CmuxNext/Sources/CmuxNextApp/Control/VideoControl.swift:35">
P2: The identity check is not bound to the file AVFoundation later opens. A pathname can be replaced after `stat`, causing `video.frames` to decode a different file; open and validate a descriptor, or decode a verified snapshot instead.</violation>
<violation number="2" location="Packages/macOS/CmuxNext/Sources/CmuxNextApp/Control/VideoControl.swift:51">
P2: Wrong-typed params fall back silently here. A fractional `side` such as 1280.5 becomes 1280, and a non-array `times` returns info with no frames and no error. Reject these with `invalid_params`, as the element checks for `times` already do. The Rust caller validates its own values, so this matters mainly for other callers.</violation>
</file>
<file name="Packages/macOS/CmuxNext/Sources/CmuxNextAgentPane/AgentVideoFrames.swift">
<violation number="1" location="Packages/macOS/CmuxNext/Sources/CmuxNextAgentPane/AgentVideoFrames.swift:88">
P2: `read` returns a report with no frames when no requested frame decodes, so `video.frames` answers success instead of `unsupported`. Return nil when `report.frames` is empty, as `AgentPaneVideoInfo.read` does. Failed decodes are also skipped with no count, so the agent cannot tell a failed frame from one that was never asked for.</violation>
<violation number="2" location="Packages/macOS/CmuxNext/Sources/CmuxNextAgentPane/AgentVideoFrames.swift:98">
P2: `luma` may read the CGContext's backing buffer after the context is released. Swift does not guarantee the context lives past its last use, so keep it alive until the bytes are copied.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| #expect(report.frames.map(\.atMs) == [200, 1600]) | ||
| let frames = try report.frames.map { try Self.image($0.data) } | ||
| #expect(frames.allSatisfy { $0.width == 320 && $0.height == 180 }) | ||
| #expect(try Self.isRed(frames[0])) |
There was a problem hiding this comment.
P2: Guard the returned frame counts with try #require before subscripting; otherwise a decode failure crashes the test process instead of reporting which expectation failed. Apply the same protection to the frames, samples, and one.frames subscripts in this suite.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Tests/CmuxNextAgentPaneTests/AgentVideoFramesTests.swift, line 88:
<comment>Guard the returned frame counts with `try #require` before subscripting; otherwise a decode failure crashes the test process instead of reporting which expectation failed. Apply the same protection to the `frames`, `samples`, and `one.frames` subscripts in this suite.</comment>
<file context>
@@ -0,0 +1,123 @@
+ #expect(report.frames.map(\.atMs) == [200, 1600])
+ let frames = try report.frames.map { try Self.image($0.data) }
+ #expect(frames.allSatisfy { $0.width == 320 && $0.height == 180 })
+ #expect(try Self.isRed(frames[0]))
+ #expect(try !Self.isRed(frames[1]))
+ // A frame is never made larger than the video.
</file context>
| kCVPixelBufferWidthKey as String: width, kCVPixelBufferHeightKey as String: height, | ||
| ]) | ||
| writer.add(input) | ||
| #expect(writer.startWriting(), "\(String(describing: writer.error))") |
There was a problem hiding this comment.
P2: A writer failure can make this loop spin forever: the helper records failures with #expect but never aborts or checks writer.status. Use throwing requirements for writer startup/appends and bound or status-check the readiness wait so codec or disk failures fail the test rather than hanging CI.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Tests/CmuxNextAgentPaneTests/AgentVideoFramesTests.swift, line 28:
<comment>A writer failure can make this loop spin forever: the helper records failures with `#expect` but never aborts or checks `writer.status`. Use throwing requirements for writer startup/appends and bound or status-check the readiness wait so codec or disk failures fail the test rather than hanging CI.</comment>
<file context>
@@ -0,0 +1,123 @@
+ kCVPixelBufferWidthKey as String: width, kCVPixelBufferHeightKey as String: height,
+ ])
+ writer.add(input)
+ #expect(writer.startWriting(), "\(String(describing: writer.error))")
+ writer.startSession(atSourceTime: .zero)
+ for index in 0..<10 {
</file context>
| /// Images a script's calls returned that one `cmux_exec` result carries | ||
| /// (a computer use screenshot), and their total base64 bytes. | ||
| const MAX_IMAGES: usize = 8; | ||
| /// (a computer use screenshot, a `video_frames` call's frames), and their |
There was a problem hiding this comment.
P2: In one script, a second video_frames call (or a call after a computer use screenshot) can exceed the shared 8 MiB image budget. Its frames then come back as dropped: true markers, and the call's text still says the frames are 'below'. Surface the script-level drop in the text, or count the video budget against the shared image budget before answering.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At cmux-tui/crates/cmux-tui/src/cli/mcp/code_mode.rs, line 994:
<comment>In one script, a second `video_frames` call (or a call after a computer use screenshot) can exceed the shared 8 MiB image budget. Its frames then come back as `dropped: true` markers, and the call's text still says the frames are 'below'. Surface the script-level drop in the text, or count the video budget against the shared image budget before answering.</comment>
<file context>
@@ -987,8 +991,9 @@ fn closed() -> Value {
/// Images a script's calls returned that one `cmux_exec` result carries
-/// (a computer use screenshot), and their total base64 bytes.
-const MAX_IMAGES: usize = 8;
+/// (a computer use screenshot, a `video_frames` call's frames), and their
+/// total base64 bytes.
+const MAX_IMAGES: usize = super::video_tools::MAX_FRAMES;
</file context>
| guard let format = AgentVideoFrames.Format(rawValue: raw) else { throw ControlError.invalidParams("video.frames format is jpeg or luma") } | ||
| return format | ||
| } ?? .jpeg | ||
| let side = params["side"]?.intValue ?? 1280 |
There was a problem hiding this comment.
P2: Wrong-typed params fall back silently here. A fractional side such as 1280.5 becomes 1280, and a non-array times returns info with no frames and no error. Reject these with invalid_params, as the element checks for times already do. The Rust caller validates its own values, so this matters mainly for other callers.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Sources/CmuxNextApp/Control/VideoControl.swift, line 51:
<comment>Wrong-typed params fall back silently here. A fractional `side` such as 1280.5 becomes 1280, and a non-array `times` returns info with no frames and no error. Reject these with `invalid_params`, as the element checks for `times` already do. The Rust caller validates its own values, so this matters mainly for other callers.</comment>
<file context>
@@ -0,0 +1,66 @@
+ guard let format = AgentVideoFrames.Format(rawValue: raw) else { throw ControlError.invalidParams("video.frames format is jpeg or luma") }
+ return format
+ } ?? .jpeg
+ let side = params["side"]?.intValue ?? 1280
+ guard AgentVideoFrames.sides.contains(side) else {
+ throw ControlError.invalidParams("video.frames side is \(AgentVideoFrames.sides.lowerBound) to \(AgentVideoFrames.sides.upperBound)")
</file context>
| space: CGColorSpaceCreateDeviceGray(), bitmapInfo: CGImageAlphaInfo.none.rawValue) else { return nil } | ||
| context.interpolationQuality = .medium | ||
| context.draw(image, in: CGRect(x: 0, y: 0, width: side, height: side)) | ||
| guard let pixels = context.data else { return nil } |
There was a problem hiding this comment.
P2: luma may read the CGContext's backing buffer after the context is released. Swift does not guarantee the context lives past its last use, so keep it alive until the bytes are copied.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Sources/CmuxNextAgentPane/AgentVideoFrames.swift, line 98:
<comment>`luma` may read the CGContext's backing buffer after the context is released. Swift does not guarantee the context lives past its last use, so keep it alive until the bytes are copied.</comment>
<file context>
@@ -0,0 +1,101 @@
+ space: CGColorSpaceCreateDeviceGray(), bitmapInfo: CGImageAlphaInfo.none.rawValue) else { return nil }
+ context.interpolationQuality = .medium
+ context.draw(image, in: CGRect(x: 0, y: 0, width: side, height: side))
+ guard let pixels = context.data else { return nil }
+ return Data(bytes: pixels, count: side * side)
+ }
</file context>
| let at = frame.actualTime.seconds.isFinite ? frame.actualTime.seconds : asked | ||
| report.frames.append(Frame(atMs: Int((min(max(at, 0), seconds) * 1000).rounded()), data: encoded)) | ||
| } | ||
| return report |
There was a problem hiding this comment.
P2: read returns a report with no frames when no requested frame decodes, so video.frames answers success instead of unsupported. Return nil when report.frames is empty, as AgentPaneVideoInfo.read does. Failed decodes are also skipped with no count, so the agent cannot tell a failed frame from one that was never asked for.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Sources/CmuxNextAgentPane/AgentVideoFrames.swift, line 88:
<comment>`read` returns a report with no frames when no requested frame decodes, so `video.frames` answers success instead of `unsupported`. Return nil when `report.frames` is empty, as `AgentPaneVideoInfo.read` does. Failed decodes are also skipped with no count, so the agent cannot tell a failed frame from one that was never asked for.</comment>
<file context>
@@ -0,0 +1,101 @@
+ let at = frame.actualTime.seconds.isFinite ? frame.actualTime.seconds : asked
+ report.frames.append(Frame(atMs: Int((min(max(at, 0), seconds) * 1000).rounded()), data: encoded))
+ }
+ return report
+ }
+
</file context>
| throw ControlError.invalidParams("video.frames needs dev and ino: the file as the caller sees it") | ||
| } | ||
| var info = stat() | ||
| guard stat(path, &info) == 0, String(UInt64(bitPattern: Int64(info.st_dev))) == dev, String(info.st_ino) == ino else { |
There was a problem hiding this comment.
P2: The identity check is not bound to the file AVFoundation later opens. A pathname can be replaced after stat, causing video.frames to decode a different file; open and validate a descriptor, or decode a verified snapshot instead.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Sources/CmuxNextApp/Control/VideoControl.swift, line 35:
<comment>The identity check is not bound to the file AVFoundation later opens. A pathname can be replaced after `stat`, causing `video.frames` to decode a different file; open and validate a descriptor, or decode a verified snapshot instead.</comment>
<file context>
@@ -0,0 +1,66 @@
+ throw ControlError.invalidParams("video.frames needs dev and ino: the file as the caller sees it")
+ }
+ var info = stat()
+ guard stat(path, &info) == 0, String(UInt64(bitPattern: Int64(info.st_dev))) == dev, String(info.st_ino) == ino else {
+ throw ControlError(code: "not_found", message: "\(path) is not a file on the Mac the cmux app runs on; a chat on another machine reads its videos with that machine's own tools")
+ }
</file context>
| fn identity(path: &str) -> Result<(String, Value), String> { | ||
| use std::os::unix::fs::MetadataExt as _; | ||
| let real = | ||
| std::fs::canonicalize(path).map_err(|error| format!("cannot read {path}: {error}"))?; |
There was a problem hiding this comment.
P3: When the file is missing on this machine, the error only says 'cannot read ...'. The module doc says remote chats are told the file is not on this Mac and to use that machine's tools, so this message should say the file is not on this machine and point to that machine's tools.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At cmux-tui/crates/cmux-tui/src/cli/mcp/video_tools.rs, line 370:
<comment>When the file is missing on this machine, the error only says 'cannot read ...'. The module doc says remote chats are told the file is not on this Mac and to use that machine's tools, so this message should say the file is not on this machine and point to that machine's tools.</comment>
<file context>
@@ -0,0 +1,781 @@
+fn identity(path: &str) -> Result<(String, Value), String> {
+ use std::os::unix::fs::MetadataExt as _;
+ let real =
+ std::fs::canonicalize(path).map_err(|error| format!("cannot read {path}: {error}"))?;
+ let meta = std::fs::metadata(&real).map_err(|error| format!("cannot read {path}: {error}"))?;
+ if !meta.is_file() {
</file context>
| text.push_str(&format!(" (change {change:.1})")); | ||
| } | ||
| } | ||
| if scenes && frames.len() == 1 { |
There was a problem hiding this comment.
P3: scenes: true, count: 1 always reports that the picture barely changes, even when scene samples contain large changes. Remove that claim or pass whether any change was detected separately from the selected frame count.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At cmux-tui/crates/cmux-tui/src/cli/mcp/video_tools.rs, line 320:
<comment>`scenes: true, count: 1` always reports that the picture barely changes, even when scene samples contain large changes. Remove that claim or pass whether any change was detected separately from the selected frame count.</comment>
<file context>
@@ -0,0 +1,781 @@
+ text.push_str(&format!(" (change {change:.1})"));
+ }
+ }
+ if scenes && frames.len() == 1 {
+ text.push_str(
+ "\nThe picture barely changes: ask for range or count frames spread over it instead.",
</file context>
|
Review (codex gpt-6.1-sol xhigh, router-review): BLOCK at af1a078 BLOCK at af1a078 Visual verdict: no images; Leo tries the build
Rust and Swift tests could not run because neither toolchain is installed. Follow-ups:
|
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
…es in video.frames Agent-Session: https://cmux.com/deeplink/agent?id=aid-bbe6e8ca1bc11f92f25b Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub Actions
Inactive
GitHub Actions
Error
…ports Foundation publicly Review (gpt-6.1-sol) at af1a078: the public read(URL) needs public import Foundation under InternalImportsByDefault, and scene sampling started half a step in (28 s into an hour). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-bbe6e8ca1bc11f92f25b
GitHub Actions
Error
GitHub Actions
Inactive
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-bbe6e8ca1bc11f92f25b
GitHub Actions
Inactive
GitHub Actions
Inactive
|
Found 4 test failures on Blacksmith runners: Failures
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
GitHub Actions
Inactive
GitHub Actions
Inactive
|
Review (codex gpt-6-astra high, router-review): BLOCK at a971d85 BLOCK at a971d85 Visual verdict: no images; Leo tries the build
Follow-ups:
|
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
Review (gpt-6-astra) at a971d85. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-bbe6e8ca1bc11f92f25b
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
![cmux-bot[bot]](https://avatars.githubusercontent.com/u/171392238?s=60&v=4){kind=small}
A newer verdict blocks b97dd1b (no independent SHIP at the head).
|
This PR added |
…d forms (#31097) #30921 (af670df, cmux mcp video_frames/video_info) and #30923 (b91b9c7, cmux link seed pull) added clippy::indexing_slicing hits in cmux-tui production code, a deny lint of the panic ratchet: cmux-tui went from 831 to 855. Each read now uses a checked form; valid input behaves the same, and a missing or non-object field reads as it did through serde_json's Index (null): - video_tools: the app's answer and error fields through Value::get; the params writes through cmux_json::JsonSet::set (the params are always an object, as before). - seed_pull export: the job and status fields through Value::get; the bundle copy's chunk through buffer.get(..read), an error if a read ever claims more than its buffer. - seed_cli pull keep: summary.base.{head,index,work} through Value::get. No allow attributes, no baseline change. Co-authored-by: cmux-bot[bot] <348531757+cmux-bot[bot]@users.noreply.github.com>
…l, RoomMembership, IncognitoSpaceMarks out of CmuxNextApp (#29184 #30921 #30666 #30881 #30828) (#31036) * base red: arch app_lines, move five App files into their packages The feat-cmux-next tip is red on the arch ratchet's app_lines row: +295 CmuxNextApp lines over the last green arch tip 73dc0d8. This moves five self-contained App files into the package that owns their domain, with no behavior change (public API, the App calls it): - CloudMoveAlwaysGrants (#29184) and CloudMoveCLI -> CmuxNextCloud/Move, beside CloudMoveFolder and CloudMoveRetries. CmuxNextCloud now depends on CmuxNextDaemon for ProcessRunner (both service layer). - VideoControl (#30921) -> CmuxNextAgentPane, beside AgentVideoFrames. A struct with an instance methods() (namespace convention); the App registers VideoControl().methods(). CmuxNextAgentPane now depends on CmuxNextControl (ui -> service). - RoomMembership (#30666 added isHidden) -> CmuxNextDaemon/Store, beside PersonalStore whose pins it reads. A pure value type. - IncognitoSpaceMarks (#30881) -> CmuxNextBridge, beside SidebarMapping+Spaces that takes its hidden set. Still MainActor (Bridge has the UI default isolation). Tests that name the moved types import their new module. * base red: arch global_mutable_statics, IncognitoChats' pending purges are the app's instance (#30828) #30828 added 'private static var pending' to the IncognitoChats enum, which raises the arch ratchet's global_mutable_statics row (+1, CmuxNextApp). IncognitoChats is now a final class that AppServices owns (services.incognitoChats); WorkspaceClose.willClose and the quit's windows and endLocalAgents steps call that one instance. The pure key(of:key:incognito:) stays static. No behavior change: one app, one set of pending purges, drained at quit as before.
Leo, 2026-11-01, after he attached a 16 s mp4 in a cmux NEXT chat and the agent had to hand-write
ffmpeg -vf fps=1,scale=640:-1,tile=4x4into /tmp to look at it: "i feel like cmux-next should give u better tools to deal with this kind of thing ... you can see what we can work on", then "can you help work on all these".What
Two new tools on the
cmuxMCP server (cmux-tuicli/mcp/video_tools.rs), listed incmux_exec's common tools so agents pick them over ffmpeg:video_info {path}: duration (ms), width and height as shown (after rotation), fps, has audio.video_frames {path, at | range | scenes, count, maxSide}: frames as MCP image blocks, plus one text block with the duration, size, fps, audio and each frame's time.at: [seconds…]: exact frames at those times (1 to 16).range: {from, to, count}:countframes evenly fromfromtoto, both included (defaults 0, the end, 8).scenes: true(withcount, default 8): the first frame plus the moments the picture changes most, in time order, each with its change score.countframes at the middles of equal parts of the whole video (default 8).maxSide: longest side in px, 64 to 2048, default 1280; never upscaled.Bounded output: at most 16 frames a call and 6 MiB of base64 JPEG. A call over the budget returns the first frames and says how many it left out, and what to ask for next.
cmux_exec's per-script image cap goes from 8 to 16 (video_tools::MAX_FRAMES) so a 16-frame call fits; its 8 MiB byte cap is unchanged.How
~/, or relative to the chat's folder, asopendoes), picks the times and the decode tolerance (exact forat, a quarter of the spacing otherwise), finds scene changes from 64 small grey samples (mean absolute difference between neighbours, a 1.5-grey-level noise floor), and builds the image blocks. The result has nostructuredContent, socmux_execmoves the images into its own result (code_mode::script_answer).video.frames(CmuxNextApp/Control/VideoControl.swift) over a newAgentVideoFrames(CmuxNextAgentPane/AgentVideoFrames.swift). It uses the composer's AVFoundation decoder and its helpers fromAgentPaneVideoInfo(isRegularFile,jpeg, andforbidAllreference restrictions). It decodes only the asked frames, never the whole file.not_foundif its ownstatof the path differs, so a chat whose MCP server runs on another machine never gets frames from a same-named file on the Mac.Remote and Cloud sessions
The decode runs in the cmux app on the Mac. In a Cloud or ssh chat the MCP server runs on the remote machine, and the call fails cleanly in one of two ways. If no app answers, the error is
app.not_foundorapp.unreachable. If the path belongs to another machine, the app answersnot_foundand the message tells the agent to use that machine's own tools. Follow-up: a decoder on the remote side (an ffmpeg/ffprobe backend in the same Rust module, picked when no local app owns the file) gives remote chats the same tool with the same schema.Overlap
Another lane is changing
AgentPaneVideoInfo.swift's frame picking (scene detection, timestamps). This PR does not touch that file. It only calls its existingisRegularFileandjpeghelpers from a new file. Scene picking for this tool lives in Rust (video_tools::scene_times), since base has no Swift scene picker. If the other lane's Swift picker lands, the tool can keep its own or switch to it later; nothing conflicts.Tests
video_tools::tests: schema and caps, argument errors, time picking and clamping, scene picking, image blocks after one text block, the probe-then-frames request shape (path, dev/ino, side, tolerance, budget), the scene two-pass flow, left-out frames, a refusal from the app, and a missing file.AgentVideoFramesTests: numbers only, frames at exact times and size (red then blue, never upscaled), grey samples that change with the picture, the byte budget, and a non-video file.Risk: additive. Two new tools, one new read-only app method, and one constant raised (8 to 16 images per
cmux_execscript, under the same 8 MiB cap).🤖 Generated with Claude Code
Agent session: agent session
Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
Adds
video_infoandvideo_framestools to thecmuxMCP server so agents can inspect a video directly instead of hand-writing ffmpeg commands.video_info {path}returns duration (ms), displayed width and height, fps, and audio presence.video_frames {path, at | range | scenes, count, maxSide}returns frames as MCP image blocks plus a text block listing each frame's time; capped at 16 frames and 6 MiB of base64 JPEG. An over-budget call returns the first frames and says how many it left out, or tells the agent to lowermaxSidewhen the first frame alone is too big.cmux_exec's per-script image cap from 8 to 16 so a 16-frame call fits; the 8 MiB byte cap is unchanged.Written for commit b97dd1b. Summary will update on new commits.