Repository navigation
feat(cli): add TUI image display tool #8217
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
DragonnZhang
merged 22 commits into
QwenLM:main
from
DragonnZhang:dragon/tui-display-image
Aug 1, 2026
Merged
Changes from 20 commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
952e94d
feat(cli): add TUI image display tool
DragonnZhang 3d8818e
fix(cli): report terminal image rendering failures
DragonnZhang 6814fe8
fix(cli): disable Kitty placeholders in Warp
DragonnZhang 2abf06b
fix(cli): constrain terminal image previews
DragonnZhang 05c11b4
fix(core): preserve fork image tool cache prefix
DragonnZhang 0975f55
Merge branch 'main' into dragon/tui-display-image
wenshao 7a86f6a
chore(core): merge main and reconcile fork execution allowlist
qwen-code-dev-bot 996656b
Merge branch 'main' into dragon/tui-display-image
qwen-code-dev-bot f19c16f
fix(cli): address display_image review feedback (#8217)
qwen-code-dev-bot 0546f1d
fix(web-shell): add display_image to tool display contract (#8217)
qwen-code-dev-bot acd1ec7
test(cli): cover display image validation edges
DragonnZhang 93e588e
Merge branch 'main' into dragon/tui-display-image
qwen-code-dev-bot 1515975
fix(cli): harden chafa executable lookup (#8217)
qwen-code-dev-bot 735e2f9
Merge branch 'main' into dragon/tui-display-image
1033739
fix(cli): address TUI image display review feedback (#8217)
7ede190
fix(cli): preserve color in chafa image fallback
DragonnZhang 8f8b984
Merge branch 'dragon/tui-display-image' of https://github.com/Dragonn…
qwen-code-dev-bot 00cf11d
feat(cli): render images directly in Warp
DragonnZhang e4921a0
fix(cli): disable native images in Warp
DragonnZhang 913923d
test(cli): cover chafa shell path guard
DragonnZhang e1a2984
fix(cli): skip re-transmitting Kitty image payload on remount (#8217)
qwen-code-dev-bot 3e568af
Merge branch 'dragon/tui-display-image' of https://github.com/Dragonn…
qwen-code-dev-bot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| # TUI Display Image Tool | ||
|
|
||
| ## Goal | ||
|
|
||
| Add a model-invocable `display_image` tool that previews an existing PNG in | ||
| the interactive terminal UI. The tool is display-only: it does not provide | ||
| image pixels to the model and is not available in headless, SDK, screen-reader, | ||
| or subagent registries. | ||
|
|
||
| ## Design | ||
|
|
||
| The core tool validates an absolute workspace path, regular-file status, PNG | ||
| signature, and an 8 MiB size limit. On success it returns a small structured | ||
| display value: | ||
|
|
||
| ```ts | ||
| { | ||
| type: 'terminal_image', | ||
| filePath: '/absolute/workspace/path/image.png', | ||
| mimeType: 'image/png', | ||
| } | ||
| ``` | ||
|
|
||
| Before returning success, the tool asks a CLI-injected provider whether the | ||
| current terminal has an available renderer. Native protocols pass immediately; | ||
| the fallback checks that `chafa` is available on `PATH` without rendering the | ||
| selected file. If no renderer is available, the tool returns an execution error | ||
| so both the TUI and the model know that the image was not displayed. | ||
| `llmContent` tells the model to use `read_file` if it needs to inspect the image. | ||
| Image bytes and terminal escape sequences never enter model history or the | ||
| persisted display value. | ||
|
|
||
| `ToolMessage` recognizes the structured value and delegates it to a TUI | ||
| component. The component revalidates the path against the current workspace | ||
| before reading it. Rendering is prepared during the component's first render | ||
| because completed tool rows enter Ink's append-only `Static` region | ||
| immediately; an asynchronous state update would be dropped there. | ||
| Kitty-compatible terminals use the existing virtual-image and | ||
| Unicode-placeholder approach already used for Mermaid diagrams. Other | ||
| terminals try `chafa` with symbol output. `chafa` receives only the same | ||
| allowlisted environment used by the Mermaid renderer, so API credentials are | ||
| not forwarded. If neither path is available, the TUI shows a bounded text | ||
| fallback naming the image. | ||
|
|
||
| Previews are capped at 72 columns and 24 rows, then reduced further to fit the | ||
| available terminal space. An 8-by-16-pixel terminal cell estimate provides a | ||
| conservative natural size so small images are not deliberately enlarged. Both | ||
| native and `chafa` rendering use the same aspect-preserving dimensions. | ||
|
|
||
| The tool is registered only when all of these are true: | ||
|
|
||
| - the main config is interactive; | ||
| - SDK mode is disabled; | ||
| - screen-reader mode is disabled; | ||
| - the registry is not being created for a subagent. | ||
|
|
||
| Bare mode keeps its existing minimal tool set. | ||
|
|
||
| ## Compatibility | ||
|
|
||
| The first version supports PNG only. Kitty and Ghostty use native image | ||
| placement. Warp is intentionally excluded because its Kitty support does not | ||
| include the Unicode-placeholder extension used by this renderer, while direct | ||
| placement drifts away from Ink content during scrollback and terminal reflow. | ||
| Direct iTerm2 placement is also excluded because its inline image protocol is | ||
| cursor-positioned and the existing asynchronous Ink path already treats it as | ||
| unsafe. Warp, iTerm2, tmux, SSH, and terminals without a native path can still | ||
| render through `chafa` when it is installed. | ||
|
|
||
| ## Security and persistence | ||
|
|
||
| Both the tool and renderer use `WorkspaceContext.isPathWithinWorkspace`, which | ||
| resolves symlinks and prevents traversal outside registered workspace roots. | ||
| The renderer treats restored `resultDisplay` data as untrusted and repeats this | ||
| check. Persisted sessions contain only the path and MIME type; if the file no | ||
| longer exists, restore degrades to a text notice. | ||
|
|
||
| Raw terminal output is generated only by the trusted renderer. Model-controlled | ||
| strings and file contents are never written directly to the terminal. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,156 @@ | ||
| /** | ||
| * @license | ||
| * Copyright 2026 Qwen Team | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import { beforeEach, describe, expect, it, vi } from 'vitest'; | ||
| import { render } from 'ink-testing-library'; | ||
| import { Static } from 'ink'; | ||
| import type { Config } from '@qwen-code/qwen-code-core'; | ||
| import { TerminalOutputProvider } from '../contexts/TerminalOutputContext.js'; | ||
| import { | ||
| renderTerminalImage, | ||
| type TerminalImageRenderResult, | ||
| } from '../utils/terminal-image-renderer.js'; | ||
| import { TerminalImage } from './TerminalImage.js'; | ||
|
|
||
| vi.mock('../utils/terminal-image-renderer.js', () => ({ | ||
| renderTerminalImage: vi.fn(), | ||
| })); | ||
|
|
||
| const mockedRenderTerminalImage = vi.mocked(renderTerminalImage); | ||
|
|
||
| function configWithWorkspaceResult(isWithinWorkspace: boolean): Config { | ||
| return { | ||
| getWorkspaceContext: () => ({ | ||
| isPathWithinWorkspace: () => isWithinWorkspace, | ||
| }), | ||
| } as unknown as Config; | ||
| } | ||
|
|
||
| const IMAGE = { | ||
| type: 'terminal_image' as const, | ||
| filePath: '/workspace/chart.png', | ||
| mimeType: 'image/png' as const, | ||
| }; | ||
|
|
||
| function renderImage( | ||
| result: TerminalImageRenderResult, | ||
| writeRaw: (...args: unknown[]) => void = vi.fn(), | ||
| ) { | ||
| mockedRenderTerminalImage.mockReturnValueOnce(result); | ||
| return render( | ||
| <TerminalOutputProvider value={writeRaw}> | ||
| <Static items={[IMAGE]}> | ||
| {(item) => ( | ||
| <TerminalImage | ||
| data={item} | ||
| config={configWithWorkspaceResult(true)} | ||
| contentWidth={80} | ||
| availableTerminalHeight={20} | ||
| /> | ||
| )} | ||
| </Static> | ||
| </TerminalOutputProvider>, | ||
| ); | ||
| } | ||
|
|
||
| describe('TerminalImage', () => { | ||
| beforeEach(() => { | ||
| vi.clearAllMocks(); | ||
| }); | ||
|
|
||
| it('writes trusted Kitty data and renders its placeholder', async () => { | ||
| const writeRaw = vi.fn(); | ||
| const { lastFrame } = renderImage( | ||
| { | ||
| kind: 'kitty', | ||
| sequence: '\x1b_Gpayload\x1b\\', | ||
| placeholder: { | ||
| color: '#00002a', | ||
| imageId: 42, | ||
| lines: ['placeholder'], | ||
| }, | ||
| }, | ||
| writeRaw, | ||
| ); | ||
|
|
||
| await vi.waitFor(() => { | ||
| expect(writeRaw).toHaveBeenCalledWith('\x1b_Gpayload\x1b\\'); | ||
| }); | ||
| expect(lastFrame()).toContain('placeholder'); | ||
| }); | ||
|
|
||
| it('renders chafa ansi output', () => { | ||
| const { lastFrame } = renderImage({ kind: 'ansi', lines: ['▀▀', '▄▄'] }); | ||
|
|
||
| expect(lastFrame()).toContain('▀▀'); | ||
| expect(lastFrame()).toContain('▄▄'); | ||
| }); | ||
|
|
||
| it('shows a readable fallback when no renderer is available', () => { | ||
| const { lastFrame } = renderImage({ | ||
| kind: 'unavailable', | ||
| reason: 'chafa is not installed', | ||
| }); | ||
|
|
||
| expect(lastFrame()).toContain('chafa is not installed'); | ||
| expect(lastFrame()).toContain('chart.png'); | ||
| }); | ||
|
|
||
| it('refuses restored paths outside the current workspace', () => { | ||
| const { lastFrame } = render( | ||
| <TerminalImage | ||
| data={{ | ||
| type: 'terminal_image', | ||
| filePath: '/outside/chart.png', | ||
| mimeType: 'image/png', | ||
| }} | ||
| config={configWithWorkspaceResult(false)} | ||
| contentWidth={80} | ||
| />, | ||
| ); | ||
|
|
||
| expect(lastFrame()).toContain('outside the current workspace'); | ||
| expect(mockedRenderTerminalImage).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| it('does not re-emit the Kitty sequence when the emit effect re-runs', async () => { | ||
| mockedRenderTerminalImage.mockReturnValue({ | ||
| kind: 'kitty', | ||
| sequence: '\x1b_Gpayload\x1b\\', | ||
| placeholder: { | ||
| color: '#00002a', | ||
| imageId: 42, | ||
| lines: ['placeholder'], | ||
| }, | ||
| }); | ||
|
|
||
| const renderWith = (writer: (...args: unknown[]) => void) => ( | ||
| <TerminalOutputProvider value={writer}> | ||
| <TerminalImage | ||
| data={IMAGE} | ||
| config={configWithWorkspaceResult(true)} | ||
| contentWidth={80} | ||
| availableTerminalHeight={20} | ||
| /> | ||
| </TerminalOutputProvider> | ||
| ); | ||
|
|
||
| const firstWriteRaw = vi.fn(); | ||
| const { rerender } = render(renderWith(firstWriteRaw)); | ||
|
|
||
| await vi.waitFor(() => { | ||
| expect(firstWriteRaw).toHaveBeenCalledWith('\x1b_Gpayload\x1b\\'); | ||
| }); | ||
| expect(firstWriteRaw).toHaveBeenCalledTimes(1); | ||
|
|
||
| const secondWriteRaw = vi.fn(); | ||
| rerender(renderWith(secondWriteRaw)); | ||
| await new Promise((resolve) => setTimeout(resolve, 10)); | ||
|
|
||
| expect(secondWriteRaw).not.toHaveBeenCalled(); | ||
| expect(firstWriteRaw).toHaveBeenCalledTimes(1); | ||
| }); | ||
| }); | ||
|
DragonnZhang marked this conversation as resolved.
Resolved
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| /** | ||
| * @license | ||
| * Copyright 2026 Qwen Team | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import path from 'node:path'; | ||
| import React from 'react'; | ||
| import { Box, Text } from 'ink'; | ||
| import type { Config, TerminalImageDisplay } from '@qwen-code/qwen-code-core'; | ||
| import { MaxSizedBox } from './shared/MaxSizedBox.js'; | ||
| import { useTerminalOutput } from '../contexts/TerminalOutputContext.js'; | ||
| import { | ||
| renderTerminalImage, | ||
| type TerminalImageRenderResult, | ||
| } from '../utils/terminal-image-renderer.js'; | ||
| import { theme } from '../semantic-colors.js'; | ||
| import { | ||
| sanitizeMultilineForDisplay, | ||
| sanitizeTerminalText, | ||
| } from '../utils/textUtils.js'; | ||
|
|
||
| interface TerminalImageProps { | ||
| data: TerminalImageDisplay; | ||
| config: Config; | ||
| contentWidth: number; | ||
| availableTerminalHeight?: number; | ||
| } | ||
|
|
||
| export const TerminalImage: React.FC<TerminalImageProps> = ({ | ||
| data, | ||
| config, | ||
| contentWidth, | ||
| availableTerminalHeight, | ||
| }) => { | ||
| const writeRaw = useTerminalOutput(); | ||
| const writtenSequence = React.useRef<string | null>(null); | ||
| const filePath = path.resolve(data.filePath); | ||
| const safePath = config.getWorkspaceContext().isPathWithinWorkspace(filePath); | ||
| const result = React.useMemo<TerminalImageRenderResult | null>( | ||
| () => | ||
| safePath | ||
| ? renderTerminalImage({ | ||
| display: { | ||
| type: 'terminal_image', | ||
| filePath, | ||
| mimeType: data.mimeType, | ||
| }, | ||
| contentWidth, | ||
| availableTerminalHeight, | ||
| }) | ||
| : null, | ||
| [availableTerminalHeight, contentWidth, data.mimeType, filePath, safePath], | ||
| ); | ||
|
|
||
| const kittySequence = result?.kind === 'kitty' ? result.sequence : null; | ||
| React.useEffect(() => { | ||
| writtenSequence.current = null; | ||
| }, [filePath]); | ||
| React.useEffect(() => { | ||
| if (!kittySequence || writtenSequence.current === kittySequence) return; | ||
| writtenSequence.current = kittySequence; | ||
| process.nextTick(() => writeRaw(kittySequence)); | ||
|
DragonnZhang marked this conversation as resolved.
Resolved
|
||
| }, [kittySequence, writeRaw]); | ||
|
|
||
| if (!safePath) { | ||
| return ( | ||
| <Text color={theme.status.error}> | ||
| Refusing to display an image outside the current workspace. | ||
| </Text> | ||
| ); | ||
| } | ||
| if (!result) return null; | ||
| if (result.kind === 'unavailable') { | ||
| const fileName = sanitizeMultilineForDisplay(path.basename(filePath)); | ||
| return ( | ||
| <Text color={theme.text.secondary} wrap="wrap"> | ||
| {fileName}: {sanitizeTerminalText(result.reason)} | ||
| </Text> | ||
| ); | ||
| } | ||
| if (result.kind === 'ansi') { | ||
| return ( | ||
| <MaxSizedBox | ||
| maxHeight={availableTerminalHeight} | ||
| maxWidth={contentWidth} | ||
| overflowDirection="bottom" | ||
| > | ||
| {result.lines.map((line, index) => ( | ||
| <Box key={index}> | ||
| <Text>{line || ' '}</Text> | ||
| </Box> | ||
| ))} | ||
| </MaxSizedBox> | ||
| ); | ||
| } | ||
| return ( | ||
| <MaxSizedBox | ||
| maxHeight={availableTerminalHeight} | ||
| maxWidth={contentWidth} | ||
| overflowDirection="bottom" | ||
| > | ||
| {result.placeholder.lines.map((line, index) => ( | ||
| <Box key={index}> | ||
| <Text color={result.placeholder.color} wrap="truncate-end"> | ||
| {line} | ||
| </Text> | ||
| </Box> | ||
| ))} | ||
| </MaxSizedBox> | ||
| ); | ||
| }; | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.