Skip to content

Chat: send images and PDFs to acpmux ahead of the prompt, by reference - #23047

Closed
teamleaderleo wants to merge 24 commits into
feat-cmux-nextfrom
chat-attachment-put
Closed

teamleaderleo wants to merge 24 commits into
feat-cmux-nextfrom
chat-attachment-put

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Images and PDFs now go to acpmux ahead of their prompt, and the prompt names them by reference, so no prompt frame carries image bytes.
  • New acpmux method _acpmux/attachment_put {sessionId, mimeType, data} keeps an image or PDF (at most 10 MiB; a larger one still goes inline) by SHA-256 under <ACPMUX_HOME>/attachments, the same store prompt images already use. It marks the file as that chat's and answers {id}.
  • A prompt block names a kept file as uri: "acpmux-attachment:<id>" with no data. acpmux puts the bytes back before anything else reads the prompt, so user_message image ids, _acpmux/attachment reads, PDF handling and harness delivery are unchanged. A prompt that names an id its own chat did not put is refused with attachment.missing; it is not run without the image.
  • Host (Swift AcpmuxPaneMethods, plus policy.json): the put is allowed only for a chat in the pane's scope, with exactly sessionId, mimeType and data, and needs no gesture. It runs nothing; the prompt still needs its gesture. The relay already strips _meta inside prompt blocks, which is why the reference uses ACP's uri field.
  • Page (direct.ts putAttachments, attachments.ts promptBlocks): it puts only when the daemon lists _acpmux/attachment_put in initialize. Images go inline as before for an older acpmux, a peer's acpmux without the method, or a put that fails.

Why

On 2026-10-20 a coordinator chat lost its connection. A prompt with six Retina screenshots was about 15 MB of base64, and the next one passed acpmux's 16 MiB WebSocket frame limit. acpmux reset the socket, and the chat said "Not sent: The agent connection was interrupted". The product call is never to refuse a person's screenshots and to send attachments out of band, so prompt size never hits a frame limit. Related PRs:

Tests (committed red before the fix)

  • cmux-tui/crates/acpmux/tests/prompt_images.rs an_attachment_put_ahead_goes_with_the_prompt_that_names_it:
    • a put answers the image's SHA-256, and a non-image type is refused;
    • a prompt naming the id runs, its user_message names the image, and _acpmux/attachment reads its bytes back;
    • another chat's prompt naming the same id is refused with attachment.missing, and nothing is logged for it.
  • AgentPaneAttachmentRelayTests.anImageIsPutOnlyForTheChatInTheShapeThePaneSends: refuses a chat outside the pane's scope and an extra param, and passes the pane's own shape without a gesture.
  • webviews/src/agent-session/acpmux/attachmentPut.test.ts:
    • promptBlocks names a kept image by reference;
    • on the mock acpmux, a prompt with two screenshots sends two puts and a prompt frame without the bytes;
    • an acpmux without the method gets the image inline;
    • a prompt sent after one still putting its images is written after it;
    • a prompt removed from the queue while its images are put is never sent (the pane withdraws it, since acpmux has no such prompt yet).
  • The mock acpmux now lists the method and expands references, so the Playwright sent-images test exercises the whole path.
  • Not run locally: Rust, Swift and bun tests are not run on this Mac. CI runs them.

Notes

  • The method is added to the Swift allowlists and the Rust mirror policy.json alike; their lists match.
  • Nothing garbage-collects attachments/ today (prompt images on base have none either); the put marks are empty files beside it. A follow-up gives puts the age sweep and purge-on-kill that hub/file_uploads.rs has.

Agent session: cosmic-gray

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Images and PDFs are sent to acpmux ahead of the prompt that names them, by URI, keeping attachment bytes out of prompt frames and avoiding WebSocket frame-limit disconnects. Prompts reach acpmux in send order, even when earlier attachment puts take longer.

  • _acpmux/attachment_put keeps images and PDFs up to 10 MiB by SHA-256 and marks them for that chat, so the base64 read-back fits one acpmux message.
  • acpmux restores referenced bytes before running the prompt and rejects attachments that chat did not put with attachment.missing.
  • Prompts wait only for their own puts and for a prior prompt whose request is not yet written; a prompt with nothing to put still writes its request in the send call, so a dropped socket fails it like any in-flight request.
  • If the method is unavailable, a file is over the cap, or a put fails or times out after 30 seconds, attachments go inline as before.
  • Withdrawing a prompt still putting its images removes it in the pane and the send drops it; a retry of the same id stays withdrawn, and acpmux drops a copy it already took.

Written for commit dd8e557. Summary will update on new commits.

View guided diff Turn on auto-fix

teamleaderleo and others added 2 commits October 20, 2026 23:16
Red before the fix: acpmux has no _acpmux/attachment_put, the host
refuses it, and the page sends every image inline in the prompt frame.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15cf0-cb72-7882-92b8-379487dc4c66&host=cosmic-gray
Six Retina screenshots in one prompt made a frame past acpmux's 16 MiB
WebSocket limit, and acpmux reset the chat's connection (2026-10-20).
Now the page puts each image or PDF with _acpmux/attachment_put
(sessionId, mimeType, data), which acpmux keeps by SHA-256 as it already
keeps prompt images, marked as that chat's. The prompt block names it
as uri: acpmux-attachment:<id> with no data, and acpmux puts the bytes
back before the prompt runs. An id that chat did not put refuses the
prompt (attachment.missing) instead of running it without the image.

The host allows the put for a chat in the pane's scope, in the pane's
shape and with no gesture: it runs nothing, and the prompt keeps its
gesture. An acpmux without the method (an older one, or a peer's), or
a put that fails, gets the images inline as before. A prompt frame is
now small, so the host's prompt journal keeps image prompts for resend
too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15cf0-cb72-7882-92b8-379487dc4c66&host=cosmic-gray
@cursor

cursor Bot commented Oct 21, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 21, 2026 06:17 — with GitHub Actions Inactive
@teamleaderleo

teamleaderleo commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review dd8e557: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23047 -f ref=chat-attachment-put -f platform=macos -f backend_mode=local

Passes: CI passes on dd8e557d27.

CI passes on dd8e557d27 (run 39795132591 attempt 1).

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

web and size checks: green

CI fast guards passes on 9bc360f5d0 (run). This covers only the fast guards, not CI: CI's result is ci-status.

Tour of 9bc360f5d0 (run): no media. The tour job failed before the tour ran (build or setup); the run log has the output.

Gallery before/after for dd8e557: 16 states rendered, 0 images differ, 1582 more touched states not rendered. Images are in the run's artifact, not on this public PR.

gh run download 39795131972 -R manaflow-ai/cmux -n gallery-advisory-23047-dd8e557d27931d0cb0147649f18cd7a28549f941 (then read gallery-review.json)

States
State Theme Status Diff Picked by
agent-pane.chat-width/standard Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.chat-width/standard Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.chat-width/wide Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.chat-width/wide Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.chat-width/wide-blocks Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.chat-width/wide-blocks Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/new-chat Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/new-chat Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/side-session Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/side-session Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/new-chat-suggest-open Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/new-chat-suggest-open Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/new-chat-suggest-failing Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/new-chat-suggest-failing Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/new-chat-suggest-draft Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/new-chat-suggest-draft Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/new-chat-reasoning Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/new-chat-reasoning Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/agent-home Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/agent-home Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/agent-home-path Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer/agent-home-path Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer-floats/scrolled-to-end Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer-floats/scrolled-to-end Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer-floats/composer-grown Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer-floats/composer-grown Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer-row/claude-auto Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer-row/claude-auto Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer-row/long-model Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer-row/long-model Apple System Colors unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer-row/leo-report Apple System Colors Light unchanged 0% imports attachments.ts (1 hops)
agent-pane.composer-row/leo-report Apple System Colors unchanged 0% imports attachments.ts (1 hops)

Pick other states with a Gallery: entry.id, entry.id/variant line in the PR body.

@teamleaderleo

teamleaderleo commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Visual preview of ca401a9: in the maintainers' private review feed.

@blacksmith-sh

blacksmith-sh Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Found 18 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmplv5ayi1p/summary-missing.json (live):/summary-missing.json (live): View Logs
/tmp/tmpyb3t49xo/summary-fail.json (live):/summary-fail.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
cmux-next-ios-kit.yml runs macOS jobs on pull requests but a new push does not cancel t
he previous run; key the concurrency group on the pull request and set cancel-in-progre
ss for its pull request events/
cmux-next-ios-kit.yml runs macOS jobs on pull requests but a new push does not cancel t
he previous run; key the concurrency group on the pull request and set cancel-in-progre
ss for its pull request events
View Logs
convention violations found (ERROR lines above)./
convention violations found (ERROR lines above).
View Logs
nightly cmux-tui resolve/nightly cmux-tui resolve View Logs
test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse
dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo
dules)/
test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse
dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo
dules)
View Logs

...and 8 more test failures. View all on Blacksmith: run 39719772293, run 39719772113, run 39719772135

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

…puts take

A prompt with images waited for its puts before its session/prompt, so a
prompt sent after it without attachments (a harness switch's held
prompts, a prompt behind a queued one) could reach acpmux first. Each
send now writes its request only after the one before it did; a put
with no answer in 30 s goes inline. The page-allowlist parity test
reads ATTACHMENT_PUT like PREWARM_METHOD.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15cf0-cb72-7882-92b8-379487dc4c66&host=cosmic-gray
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 21, 2026 06:27 — with GitHub Actions Inactive
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next guards / defaults-snapshot is red on 24 of 24 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

…ment-put

# Conflicts:
#	cmux-tui/crates/acpmux/src/hub/prompt_images.rs
#	cmux-tui/crates/acpmux/tests/prompt_images.rs
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 21, 2026 06:41 — with GitHub Actions Inactive
…ment-put

# Conflicts:
#	cmux-tui/crates/acpmux/src/server/requests.rs
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 21, 2026 06:44 — with GitHub Actions Inactive
…nd call

The ordering chain awaited the (empty) puts and the previous prompt even
when there was nothing to wait for, so session/prompt went out a few
microtasks later. A socket dropped right after the send then failed the
requests in flight before the prompt's was registered, and it waited
forever (transcript-steer-continuous: a Send now whose request the
dropped socket fails). Now a send waits only for its own puts and for a
prompt sent before it whose request is not written yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15cf0-cb72-7882-92b8-379487dc4c66&host=cosmic-gray
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 21, 2026 07:36 — with GitHub Actions Inactive
…ment-put

# Conflicts:
#	webviews/src/agent-session/acpmux/attachments.ts
@cursor

cursor Bot commented Oct 21, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 21, 2026 08:05 — with GitHub Actions Inactive
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-tui SDKs / protocol contract is red on 10 of 11 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next crash-class rules / cmux-next crash-class rules is red on 10 of 10 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

Picks up base's fix for the agent pane's wave.css styles, red at the last merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15cf0-cb72-7882-92b8-379487dc4c66&host=big-red
@cursor

cursor Bot commented Oct 22, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent, correctness first): SHIP at ad88838

Independent subagent reviews: SHIP at bd47c1e (full PR, incl. the agent pane allowlist and Rust policy mirror parity); its medium finding (a queued prompt removed while its images upload still ran) fixed in 47217d3 (failing test) and e036a81, re-reviewed SHIP; that review's two low findings fixed in 7d708aa, re-reviewed SHIP. ad88838 is a clean merge of feat-cmux-next (base's wave.css and ResetAllRetiredKeysTests fixes) that leaves this PR's files unchanged.

Agent session: big-red

@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 22, 2026 04:39 — with GitHub Actions Inactive
@teamleaderleo teamleaderleo added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 22, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next guards / tooling-tests is red on 15 of 17 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Orphaned: no activity for 14 h, and no lane holds a lease on it. The orphan sweep could not merge feat-cmux-next in: conflicts in Packages/macOS/CmuxNext/Sources/CmuxNextAgentPane/AcpmuxPaneMethods.swift, cmux-tui/crates/acpmux/src/hub/prompt_images.rs, cmux-tui/crates/cmux-agent-pane-policy/policy.json. Listed for a lane to resolve.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra xhigh, next-review): BLOCK at dd8e557

  • webviews/src/agent-session/acpmux/direct.ts:4289 — putAttachments waits up to 30 seconds per attachment before sending session/prompt. The host’s normal pane gesture expires after 30 seconds, and attachment_put does not consume or reserve it. If a put times out (the advertised inline fallback path), the subsequent prompt is rejected with transport.gesture_required; multiple stalled puts can delay it for minutes and lose the user’s prompt.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Superseded by #25730 (merged as 21f1b02). That PR sends prompt files by reference: the prompt names a finished upload of its own chat, the daemon resolves it before anything reads the prompt, and a reference outside the chat's uploads is refused. Unlike this PR it has no 10 MiB limit, and it covers every file type, not just images and PDFs. The app sender is being built against its contract (cmux-tui/crates/acpmux/docs/prompt-uploads.md), with resumable and binary uploads in #25793 and #25803. Nothing here is still needed, so I'm closing this rather than re-cutting it.

This branch was previously deployed

4 inactive deployments
ci-routing-agent — dd8e557d Deployed Oct 22, 2026 by github-actions[bot]
artifacts — dd8e557d Deployed Oct 22, 2026 by teamleaderleo
sentry-dsyms — dd8e557d Deployed Oct 22, 2026 by teamleaderleo
cmux-tui-merge-tree — dd8e557d Deployed Oct 22, 2026 by teamleaderleo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant