Skip to content
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
952e94d
feat(cli): add TUI image display tool
DragonnZhang Jul 31, 2026
3d8818e
fix(cli): report terminal image rendering failures
DragonnZhang Jul 31, 2026
6814fe8
fix(cli): disable Kitty placeholders in Warp
DragonnZhang Jul 31, 2026
2abf06b
fix(cli): constrain terminal image previews
DragonnZhang Jul 31, 2026
05c11b4
fix(core): preserve fork image tool cache prefix
DragonnZhang Jul 31, 2026
0975f55
Merge branch 'main' into dragon/tui-display-image
wenshao Jul 31, 2026
7a86f6a
chore(core): merge main and reconcile fork execution allowlist
qwen-code-dev-bot Jul 31, 2026
996656b
Merge branch 'main' into dragon/tui-display-image
qwen-code-dev-bot Jul 31, 2026
f19c16f
fix(cli): address display_image review feedback (#8217)
qwen-code-dev-bot Jul 31, 2026
0546f1d
fix(web-shell): add display_image to tool display contract (#8217)
qwen-code-dev-bot Jul 31, 2026
acd1ec7
test(cli): cover display image validation edges
DragonnZhang Jul 31, 2026
93e588e
Merge branch 'main' into dragon/tui-display-image
qwen-code-dev-bot Jul 31, 2026
1515975
fix(cli): harden chafa executable lookup (#8217)
qwen-code-dev-bot Aug 1, 2026
735e2f9
Merge branch 'main' into dragon/tui-display-image
Aug 1, 2026
1033739
fix(cli): address TUI image display review feedback (#8217)
Aug 1, 2026
7ede190
fix(cli): preserve color in chafa image fallback
DragonnZhang Aug 1, 2026
8f8b984
Merge branch 'dragon/tui-display-image' of https://github.com/Dragonn…
qwen-code-dev-bot Aug 1, 2026
00cf11d
feat(cli): render images directly in Warp
DragonnZhang Aug 1, 2026
e4921a0
fix(cli): disable native images in Warp
DragonnZhang Aug 1, 2026
913923d
test(cli): cover chafa shell path guard
DragonnZhang Aug 1, 2026
e1a2984
fix(cli): skip re-transmitting Kitty image payload on remount (#8217)
qwen-code-dev-bot Aug 1, 2026
3e568af
Merge branch 'dragon/tui-display-image' of https://github.com/Dragonn…
qwen-code-dev-bot Aug 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions docs/design/tui-display-image-tool.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# TUI Display Image Tool

## Goal

Add a model-invocable `display_image` tool that previews an existing PNG in
the interactive terminal UI. The tool is display-only: it does not provide
image pixels to the model and is not available in headless, SDK, screen-reader,
or subagent registries.

## Design

The core tool validates an absolute workspace path, regular-file status, PNG
signature, and an 8 MiB size limit. On success it returns a small structured
display value:

```ts
{
type: 'terminal_image',
filePath: '/absolute/workspace/path/image.png',
mimeType: 'image/png',
}
```

Before returning success, the tool asks a CLI-injected provider whether the
current terminal has an available renderer. Native protocols pass immediately;
the fallback checks that `chafa` is available on `PATH` without rendering the
selected file. If no renderer is available, the tool returns an execution error
so both the TUI and the model know that the image was not displayed.
`llmContent` tells the model to use `read_file` if it needs to inspect the image.
Image bytes and terminal escape sequences never enter model history or the
persisted display value.

`ToolMessage` recognizes the structured value and delegates it to a TUI
component. The component revalidates the path against the current workspace
before reading it. Rendering is prepared during the component's first render
because completed tool rows enter Ink's append-only `Static` region
immediately; an asynchronous state update would be dropped there.
Kitty-compatible terminals use the existing virtual-image and
Unicode-placeholder approach already used for Mermaid diagrams. Other
terminals try `chafa` with symbol output. `chafa` receives only the same
allowlisted environment used by the Mermaid renderer, so API credentials are
not forwarded. If neither path is available, the TUI shows a bounded text
fallback naming the image.

Previews are capped at 72 columns and 24 rows, then reduced further to fit the
available terminal space. An 8-by-16-pixel terminal cell estimate provides a
conservative natural size so small images are not deliberately enlarged. Both
native and `chafa` rendering use the same aspect-preserving dimensions.

The tool is registered only when all of these are true:

- the main config is interactive;
- SDK mode is disabled;
- screen-reader mode is disabled;
- the registry is not being created for a subagent.

Bare mode keeps its existing minimal tool set.

## Compatibility

The first version supports PNG only. Kitty and Ghostty use native image
placement. Warp is intentionally excluded because its Kitty support does not
include the Unicode-placeholder extension used by this renderer, while direct
placement drifts away from Ink content during scrollback and terminal reflow.
Direct iTerm2 placement is also excluded because its inline image protocol is
cursor-positioned and the existing asynchronous Ink path already treats it as
unsafe. Warp, iTerm2, tmux, SSH, and terminals without a native path can still
render through `chafa` when it is installed.

## Security and persistence

Both the tool and renderer use `WorkspaceContext.isPathWithinWorkspace`, which
resolves symlinks and prevents traversal outside registered workspace roots.
The renderer treats restored `resultDisplay` data as untrusted and repeats this
check. Persisted sessions contain only the path and MIME type; if the file no
longer exists, restore degrades to a text notice.

Raw terminal output is generated only by the trusted renderer. Model-controlled
strings and file contents are never written directly to the terminal.
8 changes: 8 additions & 0 deletions packages/cli/src/config/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2106,6 +2106,14 @@ export async function loadCliConfig(
disabledSlashCommands.length > 0 ? disabledSlashCommands : undefined,
disabledSkillNamesProvider:
bareMode || safeMode ? undefined : disabledSkillNamesProvider,
terminalImageRenderSupportProvider: interactive
? async () => {
const { getTerminalImageRenderSupport } = await import(
'../ui/utils/terminal-image-renderer.js'
);
return getTerminalImageRenderSupport();
}
: undefined,
disabledSkillLevels:
bareMode || safeMode || !Array.isArray(settings.skills?.disabledLevels)
? undefined
Expand Down
1 change: 1 addition & 0 deletions packages/cli/src/i18n/locales/en.js
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,7 @@ export default {
'toolDisplayName.Agent': 'toolDisplayName.Agent',
'toolDisplayName.Artifact': 'toolDisplayName.Artifact',
'toolDisplayName.RecordArtifact': 'toolDisplayName.RecordArtifact',
'toolDisplayName.DisplayImage': 'toolDisplayName.DisplayImage',
'toolDisplayName.Skill': 'toolDisplayName.Skill',
'toolDisplayName.EnterPlanMode': 'toolDisplayName.EnterPlanMode',
'toolDisplayName.ExitPlanMode': 'toolDisplayName.ExitPlanMode',
Expand Down
1 change: 1 addition & 0 deletions packages/cli/src/i18n/locales/zh-TW.js
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,7 @@ export default {
'toolDisplayName.Agent': 'Agent',
'toolDisplayName.Artifact': '製品',
'toolDisplayName.RecordArtifact': '記錄製品',
'toolDisplayName.DisplayImage': '顯示圖片',
'toolDisplayName.Skill': '技能',
'toolDisplayName.EnterPlanMode': '進入計畫模式',
'toolDisplayName.ExitPlanMode': '退出計畫模式',
Expand Down
1 change: 1 addition & 0 deletions packages/cli/src/i18n/locales/zh.js
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,7 @@ export default {
'toolDisplayName.Agent': 'Agent',
'toolDisplayName.Artifact': '制品',
'toolDisplayName.RecordArtifact': '记录制品',
'toolDisplayName.DisplayImage': '显示图片',
'toolDisplayName.Skill': '技能',
'toolDisplayName.EnterPlanMode': '进入计划模式',
'toolDisplayName.ExitPlanMode': '退出计划模式',
Expand Down
156 changes: 156 additions & 0 deletions packages/cli/src/ui/components/TerminalImage.test.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/

import { beforeEach, describe, expect, it, vi } from 'vitest';
import { render } from 'ink-testing-library';
import { Static } from 'ink';
import type { Config } from '@qwen-code/qwen-code-core';
import { TerminalOutputProvider } from '../contexts/TerminalOutputContext.js';
import {
renderTerminalImage,
type TerminalImageRenderResult,
} from '../utils/terminal-image-renderer.js';
import { TerminalImage } from './TerminalImage.js';

vi.mock('../utils/terminal-image-renderer.js', () => ({
renderTerminalImage: vi.fn(),
}));

const mockedRenderTerminalImage = vi.mocked(renderTerminalImage);

function configWithWorkspaceResult(isWithinWorkspace: boolean): Config {
return {
getWorkspaceContext: () => ({
isPathWithinWorkspace: () => isWithinWorkspace,
}),
} as unknown as Config;
}

const IMAGE = {
type: 'terminal_image' as const,
filePath: '/workspace/chart.png',
mimeType: 'image/png' as const,
};

function renderImage(
result: TerminalImageRenderResult,
writeRaw: (...args: unknown[]) => void = vi.fn(),
) {
mockedRenderTerminalImage.mockReturnValueOnce(result);
return render(
<TerminalOutputProvider value={writeRaw}>
<Static items={[IMAGE]}>
{(item) => (
<TerminalImage
data={item}
config={configWithWorkspaceResult(true)}
contentWidth={80}
availableTerminalHeight={20}
/>
)}
</Static>
</TerminalOutputProvider>,
);
}

describe('TerminalImage', () => {
beforeEach(() => {
vi.clearAllMocks();
});

it('writes trusted Kitty data and renders its placeholder', async () => {
const writeRaw = vi.fn();
const { lastFrame } = renderImage(
{
kind: 'kitty',
sequence: '\x1b_Gpayload\x1b\\',
placeholder: {
color: '#00002a',
imageId: 42,
lines: ['placeholder'],
},
},
writeRaw,
);

await vi.waitFor(() => {
expect(writeRaw).toHaveBeenCalledWith('\x1b_Gpayload\x1b\\');
});
expect(lastFrame()).toContain('placeholder');
});

it('renders chafa ansi output', () => {
const { lastFrame } = renderImage({ kind: 'ansi', lines: ['▀▀', '▄▄'] });

expect(lastFrame()).toContain('▀▀');
expect(lastFrame()).toContain('▄▄');
});

it('shows a readable fallback when no renderer is available', () => {
const { lastFrame } = renderImage({
kind: 'unavailable',
reason: 'chafa is not installed',
});

expect(lastFrame()).toContain('chafa is not installed');
expect(lastFrame()).toContain('chart.png');
});

it('refuses restored paths outside the current workspace', () => {
const { lastFrame } = render(
<TerminalImage
Comment thread
DragonnZhang marked this conversation as resolved.
Resolved
data={{
type: 'terminal_image',
filePath: '/outside/chart.png',
mimeType: 'image/png',
}}
config={configWithWorkspaceResult(false)}
contentWidth={80}
/>,
);

expect(lastFrame()).toContain('outside the current workspace');
expect(mockedRenderTerminalImage).not.toHaveBeenCalled();
});

it('does not re-emit the Kitty sequence when the emit effect re-runs', async () => {
mockedRenderTerminalImage.mockReturnValue({
kind: 'kitty',
sequence: '\x1b_Gpayload\x1b\\',
placeholder: {
color: '#00002a',
imageId: 42,
lines: ['placeholder'],
},
});

const renderWith = (writer: (...args: unknown[]) => void) => (
<TerminalOutputProvider value={writer}>
<TerminalImage
data={IMAGE}
config={configWithWorkspaceResult(true)}
contentWidth={80}
availableTerminalHeight={20}
/>
</TerminalOutputProvider>
);

const firstWriteRaw = vi.fn();
const { rerender } = render(renderWith(firstWriteRaw));

await vi.waitFor(() => {
expect(firstWriteRaw).toHaveBeenCalledWith('\x1b_Gpayload\x1b\\');
});
expect(firstWriteRaw).toHaveBeenCalledTimes(1);

const secondWriteRaw = vi.fn();
rerender(renderWith(secondWriteRaw));
await new Promise((resolve) => setTimeout(resolve, 10));

expect(secondWriteRaw).not.toHaveBeenCalled();
expect(firstWriteRaw).toHaveBeenCalledTimes(1);
});
});
Comment thread
DragonnZhang marked this conversation as resolved.
Resolved
112 changes: 112 additions & 0 deletions packages/cli/src/ui/components/TerminalImage.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/

import path from 'node:path';
import React from 'react';
import { Box, Text } from 'ink';
import type { Config, TerminalImageDisplay } from '@qwen-code/qwen-code-core';
import { MaxSizedBox } from './shared/MaxSizedBox.js';
import { useTerminalOutput } from '../contexts/TerminalOutputContext.js';
import {
renderTerminalImage,
type TerminalImageRenderResult,
} from '../utils/terminal-image-renderer.js';
import { theme } from '../semantic-colors.js';
import {
sanitizeMultilineForDisplay,
sanitizeTerminalText,
} from '../utils/textUtils.js';

interface TerminalImageProps {
data: TerminalImageDisplay;
config: Config;
contentWidth: number;
availableTerminalHeight?: number;
}

export const TerminalImage: React.FC<TerminalImageProps> = ({
data,
config,
contentWidth,
availableTerminalHeight,
}) => {
const writeRaw = useTerminalOutput();
const writtenSequence = React.useRef<string | null>(null);
const filePath = path.resolve(data.filePath);
const safePath = config.getWorkspaceContext().isPathWithinWorkspace(filePath);
const result = React.useMemo<TerminalImageRenderResult | null>(
() =>
safePath
? renderTerminalImage({
display: {
type: 'terminal_image',
filePath,
mimeType: data.mimeType,
},
contentWidth,
availableTerminalHeight,
})
: null,
[availableTerminalHeight, contentWidth, data.mimeType, filePath, safePath],
);

const kittySequence = result?.kind === 'kitty' ? result.sequence : null;
React.useEffect(() => {
writtenSequence.current = null;
}, [filePath]);
React.useEffect(() => {
if (!kittySequence || writtenSequence.current === kittySequence) return;
writtenSequence.current = kittySequence;
process.nextTick(() => writeRaw(kittySequence));
Comment thread
DragonnZhang marked this conversation as resolved.
Resolved
}, [kittySequence, writeRaw]);

if (!safePath) {
return (
<Text color={theme.status.error}>
Refusing to display an image outside the current workspace.
</Text>
);
}
if (!result) return null;
if (result.kind === 'unavailable') {
const fileName = sanitizeMultilineForDisplay(path.basename(filePath));
return (
<Text color={theme.text.secondary} wrap="wrap">
{fileName}: {sanitizeTerminalText(result.reason)}
</Text>
);
}
if (result.kind === 'ansi') {
return (
<MaxSizedBox
maxHeight={availableTerminalHeight}
maxWidth={contentWidth}
overflowDirection="bottom"
>
{result.lines.map((line, index) => (
<Box key={index}>
<Text>{line || ' '}</Text>
</Box>
))}
</MaxSizedBox>
);
}
return (
<MaxSizedBox
maxHeight={availableTerminalHeight}
maxWidth={contentWidth}
overflowDirection="bottom"
>
{result.placeholder.lines.map((line, index) => (
<Box key={index}>
<Text color={result.placeholder.color} wrap="truncate-end">
{line}
</Text>
</Box>
))}
</MaxSizedBox>
);
};
Loading
Loading