Show the images and PDFs an agent's tools return or write inline in the chat

[teamleaderleo]

Summary

Image outputs from an agent's tool calls now show inline in the chat, and a PDF an agent writes or links shows as a thumbnail of its first page. Before this, the transcript dropped returned images and showed a PDF only as a file chip.

This PR now includes #18357 (closed into this one), so it lands both changes:

• Returned images (from cmux-next agent pane: tool calls show the images they return or write #18357): a tool that returns an image, such as a browser screenshot, shows it. ACP image content blocks are kept as data URLs, png/jpeg/gif/webp only, up to 8 MB each.

• Written images (from cmux-next agent pane: tool calls show the images they return or write #18357): a tool that writes or prints an image file shows that file: the edited file, the call's locations, and absolute image paths in its output. They load through the host like a reply's local image (image.load), so only files inside the session's folders show. At most four per call, each capped at 360x220 and decoded lazily. A click opens the image viewer (Agent pane: image viewer for chat images #18200), as do reply images, through the shared OpenableImage.

• Host-drawn page: the host draws page 1 of the PDF with CoreGraphics, on white, with the longest side at 1024 px. It hands the pane a PNG data URL through the existing image.load path. The page never receives the document and keeps its img-src data: CSP.

• Limits: only page 1 is decoded, so PDFs up to 100 MB qualify (other images stay at 10 MB). A file that CoreGraphics can't read as a PDF is refused, as is a locked one. As for any reply image, only files inside the session's folders load.

• Tool outputs: a .pdf a tool edits, lists in its locations or prints as an absolute path now joins the tool's image strip. A reply link to a local PDF shows the same thumbnail through ReplyImage.

• Click: opens the in-pane image viewer (Agent pane: image viewer for chat images #18200), never a browser tab.

Targets feat-cmux-next and carries #18357's commits. Part of bead cx-ncc.51. Video and artifact links follow in separate PRs.

Testing

• cmux-next agent pane: tool calls show the images they return or write #18357's part: red commit 5c3dbb8, then the fix; toolImages.test.tsx covers returned and written images.
• Red commit 788b662, then the fix.
• bun test src/agent-session/acpmux/conversation/toolImages.test.tsx (local): 3 pass, 1 fail at the red commit; 4 pass after the fix.
• AgentPaneReplyImageTests.aPDFShowsItsFirstPageAsAPNG checks a two-page portrait PDF. The result must be a PNG data URL whose longest side is thumbnailSide and which stays portrait, and a fake .pdf must be refused. Fleet Swift suite (step 18909ba90b44, run-swift-testing-suites.sh at c24deb9, which carries this PR's Swift unchanged): passes.
• Review follow-up: a page smaller than the thumbnail was drawn at its natural size in white margins, because CoreGraphics' page transform never scales up. Red commit 6c94740 samples pixels on a Letter page and adds a /Rotate 90 page. Fleet suite at 6c94740: aPDFShowsItsFirstPageAsAPNG fails with 2 issues, one on each page. The fix 8eb5524 builds the transform by hand and clips to the crop box. At 8eb5524 the rotated page passes, but the Letter-page check still failed: that page stores its blue as sRGB, and its device-RGB value missed the exact-byte threshold. 476b12d matches blue by hue, and the test passes on the fleet at 476b12d.
• vp check and vp fmt pass on the changed files. verify-local.py --only swift-syntax passes on the changed Swift file. The agent pane bundle is no longer committed (base dropped it); builds generate it.

Changelog

Added: Images an agent's tools return or save (screenshots, charts) show inline in the agent chat, and a click zooms them
Added: A PDF an agent writes or links shows a first-page thumbnail in the agent chat, and a click zooms it

Proof

Before (feat-cmux-next): a tool call shows only text; returned and written images are dropped, and a PDF shows only as a file chip. The base gallery has no tool-images variant: /live/?entry=agent-pane.transcript

After: variant tool-images shows a sign-in page screenshot a browser tool returned, a build-time chart a script saved, and a three-page build-report PDF a script wrote, shown as its real first page. Preview: /wt/feat-cmux-next-media-thumbnails/?entry=agent-pane.transcript#/agent-pane.transcript/tool-images

The gallery host's branch sync is stuck on a deleted branch, so the preview shows the newest build before the latest base merge. The variant's content is the same.

🤖 Generated with Claude Code

---

Note

Medium Risk
Adds host-side PDF decoding and new transcript UI for tool outputs; bounded by size caps and session-folder rules, but CoreGraphics PDF handling is a new attack surface for malicious files.

Overview
Tool calls in the agent transcript now show inline image thumbnails under each row (even when collapsed): ACP image content blocks are kept as capped data URLs, and absolute image/PDF paths from edits, locations, or output text load through the existing host image.load path via ReplyImage, at most four per call.

On macOS, local PDFs (replies and tool outputs) no longer pass through the raster pipeline—the host renders page 1 with CoreGraphics to a PNG data URL (1024px longest side, up to 100 MB), including /Rotate, so the webview still only sees data: images under CSP.

OpenableImage centralizes click-to-zoom for shown images (markdown inline, reply images, and tool strips); gallery adds a tool-images transcript variant and tests cover PDF thumbnails and toolImages merging.

^{Reviewed by Cursor Bugbot for commit 654801f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 51610c2c-ee8e-4ccc-9a60-6362c73c0a9e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[blacksmith-sh]

Blacksmith runners detected OOM events on the following jobs:

Job	Details
web / react-apps-check	View Job

[github-actions]

CI fast guards failed on 476b12d758 (https://github.com/manaflow-ai/cmux/actions/runs/37675335094). It does not block the merge; a red guard merged into main breaks it for every open PR.

Check that push and pull_request path filters agree

Test: python3 tests/test_ci_workflow_path_filter_parity.py

Traceback (most recent call last):
  File "/home/runner/_work/cmux/cmux/tests/test_ci_workflow_path_filter_parity.py", line 247, in <module>
    test_every_push_only_pattern_is_still_needed()
  File "/home/runner/_work/cmux/cmux/tests/test_ci_workflow_path_filter_parity.py", line 183, in test_every_push_only_pattern_is_still_needed
    assert not stale, (
           ^^^^^^^^^
AssertionError: cmux-next.yml: Packages/Shared/**, Packages/iOS/** is no longer push-only. Remove it from PUSH_ONLY_PATTERNS.

Fix: the assertion above names what the guard expects; change the tree to match it.
Reproduce in seconds, no build: scripts/ci/guards-local.sh --step 'Check that push and pull_request path filters agree'

Validate package test selection

Test: python3 tests/test_ci_select_package_tests.py

    result, count = run_package_step("CmuxUpdater", [(startup, 1), (passed, 0)])
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/_work/cmux/cmux/tests/test_ci_select_package_tests.py", line 102, in run_package_step
    count = int((root / "calls").read_text())
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/pathlib.py", line 1029, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/pathlib.py", line 1015, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpiijicu32/calls'

Fix: the assertion above names what the guard expects; change the tree to match it.
Reproduce in seconds, no build: scripts/ci/guards-local.sh --step 'Validate package test selection'

Validate local hook trust boundary

Test: python3 tests/test_preflight_trust.py

• PreflightTrustTests.test_hook_installation_has_no_candidate_code_runner

AssertionError: True is not false

Fix: the assertion above names what the guard expects; change the tree to match it.
Reproduce in seconds, no build: scripts/ci/guards-local.sh --step 'Validate local hook trust boundary'

Validate the package test lane's single-suite phase

suite run failed (1): ::error title=No macOS SDK::package-test-lane.sh: xcrun found no macOS SDK in /Applications/Xcode_26.6.app/Contents/Developer (got '')

no build of the package: ''

no filtered test run: ''

string catalogs were not compiled in the package: ''

... and 11 more failing tests in the run log.
Fix: the assertion above names what the guard expects; change the tree to match it.
Reproduce in seconds, no build: scripts/ci/guards-local.sh --step 'Validate the package test lane'"'"'s single-suite phase'

Validate fork runner routing

Test: python3 tests/test_ci_fork_runner_routing.py

• ForkRunnerRoutingTests.test_fork_pull_requests_into_manaflow_ai_never_reach_an_owned_runner_label

AssertionError: Lists differ: ['cmux-next.yml:954 has no top-level fork [6389 chars] }}"] != []

First list contains 11 additional elements.
First extra element 0:
'cmux-next.yml:954 has no top-level fork pull-request branch: runs-on: ${{ needs.macos-placement.outputs.runner || github.repository_owner != \'manaflow-ai\' && \'macos-26\' || (github.event_name == \'pull_request\' && github.event.pull_request.head.repo.full_name != github.repository && \'blacksmith-6vcpu-macos-26\' || vars.CI_PR_POOL_OWNED == \'1\' && contains(fromJSON(\'["pull_request","push","workflow_dispatch"]\'), github.event_name) && (github.run_attempt <= 2 || github.triggering_actor != \'github-actions[bot]\') && vars.CI_SIDE_LANE_RUNNER || vars.MACOS_RUNNER_26 || \'blacksmith-6vcpu-macos-26\') }}'

Diff is 6836 characters long. Set self.maxDiff to None to see it.

• ForkRunnerRoutingTests.test_no_workflow_falls_back_to_blacksmith_outside_manaflow_ai

AssertionError: Lists differ: ["cmux-next.yml:954: blacksmith-6vcpu-maco[2411 chars] }}"] != []

First list contains 10 additional elements.
First extra element 0:
"cmux-next.yml:954: blacksmith-6vcpu-macos-26 is selectable outside manaflow-ai, where no Blacksmith runner exists; start the expression with the owner fork branch, e.g. ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || ... }}"

Diff is 2673 characters long. Set self.maxDiff to None to see it. : 
cmux-next.yml:954: blacksmith-6vcpu-macos-26 is selectable outside manaflow-ai, where no Blacksmith runner exists; start the expression with the owner fork branch, e.g. ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || ... }}
cmux-next.yml:962: blacksmith-6vcpu-macos-26 is selectable outside manaflow-ai, where no Blacksmith runner exists; start the expression with the owner fork branch, e.g. ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || ... }}
cmux-next.yml:1175: blacksmith-6vcpu-macos-26 is selectable outside manaflow-ai, where no Blacksmith runner exists; start the expression with the owner fork branch, e.g. ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || ... }}
cmux-next.yml:1184: blacksmith-6vcpu-macos-26 is selectable outside manaflow-ai, where no Blacksmith runner exists; start the expression with the owner fork branch, e.g. ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || ... }}
cmux-next.yml:1345: blacksmith-6vcpu-macos-26 is sel ...

Fix: the assertion above names what the guard expects; change the tree to match it.
Reproduce in seconds, no build: scripts/ci/guards-local.sh --step 'Validate fork runner routing'

Validate reusable guard workflow structure

• Wiring.test_main_full_suite_admission_compiles_on_the_seed_pool_and_xcode

AssertionError: 'blacksmith-6vcpu-macos-15' != 'pool-pr'
- blacksmith-6vcpu-macos-15
+ pool-pr

• Wiring.test_no_retry_attempt_of_any_workflow_takes_an_owned_label (workflow='cmux-next-batch.yml', job='regen', attempt='3')

AssertionError: True is not false : ['glaeda-side-std-xcode-26.6']

• Wiring.test_no_retry_attempt_of_any_workflow_takes_an_owned_label (workflow='cmux-next-batch.yml', job='build', attempt='3')

AssertionError: True is not false : ['glaeda-side-std-xcode-26.6']

Fix: the assertion above names what the guard expects; change the tree to match it.
Reproduce in seconds, no build: scripts/ci/guards-local.sh --step 'Validate reusable guard workflow structure'

Run canonical CMUX CI guard profile

test_cmux_next_does_not_run_full_suite_for_unrelated_packages

Fix: the assertion above names what the guard expects; change the tree to match it.
Reproduce in seconds, no build: scripts/ci/guards-local.sh --step 'Run canonical CMUX CI guard profile'

Agents: python3 scripts/ci/guard_attribution.py fix applies the mechanical fixes locally. This comment is updated in place on each push.

[github-actions]

Unclear: web / react-apps-check failed: no known signature; failed step: Test webviews.

CI failed on 654801fecc (run 37685720481 attempt 1): 1 unknown.
macOS jobs did not run: compile and app tests were skipped: Fast static checks skipped.

Job	Verdict	Why
web / react-apps-check	unknown	no known signature; failed step: Test webviews

Not re-run automatically: web / react-apps-check is not a machine failure.

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

[github-actions]

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/37689519735 at 3e0a1ad failed: cmux-next generated files.
Those jobs last passed at a14deb7. This pull request is one of 4 merged since: #18399, #18200, #18401, #18431.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.