feat(cli): paste base64 / data URL images, drag image files, with [Image #N] placeholders - #104
Open
BingqingLyu wants to merge 1 commit into
Open
BingqingLyu wants to merge 1 commit into
BingqingLyu wants to merge 1 commit into
Conversation
…age #N] placeholders Adds three new ways to attach images to the prompt, all converging on a shared [Image #N] placeholder UX inserted at the cursor: - Paste a `data:image/...;base64,...` URL as text. Decoded after a conservative magic-byte check. - Paste raw base64. Only accepted when the decoded prefix matches a known image magic (PNG/JPEG/GIF/WebP/BMP/TIFF) so JWTs and regular base64-looking text aren't hijacked. - Drag an image file into the terminal. Supported both when the terminal wraps the drop in bracketed paste AND when it synthesizes the drop as a rapid burst of individual keystrokes (macOS Terminal.app), via a simple keystroke-burst heuristic (>= 4 chars with < 10 ms gaps) + a debounced scan of the buffer tail. The existing Cmd+V clipboard path now also renders as [Image #N] for consistency; attachments added without a placeholder fall back to the legacy [filename] chip so other callers are unchanged. At submit time, [Image #N] tokens in the message text are substituted with @<relative path> so the model sees each image exactly where the user placed it. Counter resets on submit. Files ----- - clipboardUtils.ts: add `tryDecodeBase64Image`, `saveDecodedImage`, `detectDraggedImagePath` (path is validated to exist, be a regular file, and have a recognized image extension). Accepts single-/double-quoted paths and escaped spaces as used by drag-drop. - InputPrompt.tsx: unified paste branch (pasteImage -> base64 -> drag-drop paste -> large paste -> small paste), keystroke-burst detection for drag-drops that arrive as typing, [Image #N] allocator, attachment placeholder substitution on submit. - Tests: 143/143 pass (1 Windows skip). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
| it('accepts escaped spaces (terminal drag-drop style)', async () => { | ||
| const imagePath = path.join(tmp, 'a b.png'); | ||
| await fs.writeFile(imagePath, PNG_MAGIC); | ||
| const escaped = imagePath.replace(/ /g, '\\ '); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds three new ways to attach images to the prompt and unifies them with the existing Cmd+V path under a single
[Image #N]placeholder UX.Closes QwenLM#3518
What's new
data:image/<type>;base64,<payload>(devtools "Copy as data URL", chat messages containing embedded images, etc.).< 10 mskeystroke-gap heuristic + a 150 ms debounced scan ofbuffer.text's trailing token.Unified UX
All four sources — Cmd+V binary clipboard (existing), data URL, raw base64, drag-drop — converge on:
[Image #N]placeholder inserted at the cursor.[Image #N]→@<relative path>substitution at submit time so the model sees each image exactly where the user placed it.Attachments added without a placeholder (legacy code paths) keep rendering as
[filename], so other callers are unchanged.Code
clipboardUtils.ts:tryDecodeBase64Image(text)— data URL + raw base64, magic-byte sniff.saveDecodedImage(buf, ext, dir)— persists decoded bytes to the shared clipboard temp dir.detectDraggedImagePath(text)— validates a token is an existing local image file; supports single/double-quoted paths and escaped spaces.clipboardHasImage/saveClipboardImage/cleanupOldClipboardImagesunchanged.InputPrompt.tsx:pasteImage→ data URL / raw base64 → drag-drop paste path → large paste placeholder → small paste.DRAG_BURST_MAX_INTERVAL_MS = 10,DRAG_MIN_BURST_CHARS = 4,DRAG_CHECK_DEBOUNCE_MS = 150).[Image #N]allocator +[Image #N]→@pathsubstitution inhandleSubmitAndClear.Test plan
npm test --workspace packages/cli -- clipboardUtils InputPrompt— 143 pass, 1 Windows-skip, 0 failtryDecodeBase64Image: data URL, declared-vs-sniffed MIME (magic wins), raw base64 PNG, raw base64 JPEG, rejects text / JWT-like / empty / short / non-base64 data URL.saveDecodedImageround-trips bytes to disk.detectDraggedImagePath: existing file, quoted path, escaped-space path, non-image extension, missing file, directory, empty input, multi-line.InputPrompt > base64 / data URL paste: single paste, sequential[Image #1]/[Image #2], fall-through to large-paste placeholder for non-images.InputPrompt > drag-and-drop image paste: image path →[Image #1], non-image path → regular buffer handling.npm run build --workspace packages/clipasses.Backwards compatibility
clipboardHasImage,saveClipboardImage,cleanupOldClipboardImages).[filename]when no placeholder is set.@pathmentions outside the image paste flow.Notes
@teddyzhu/clipboardbinding issues on some platforms (Clipboard image paste (Cmd+V) silently fails on macOS — two root causes QwenLM/qwen-code#3517). This PR leaves that path untouched.