Skip to content

feat(cli): paste base64 / data URL images, drag image files, with [Image #N] placeholders - #104

Open
BingqingLyu wants to merge 1 commit into
mainfrom
fork-pr-3519-feat-base64-image-paste
Open

BingqingLyu wants to merge 1 commit into
mainfrom
fork-pr-3519-feat-base64-image-paste

Conversation

@BingqingLyu

@BingqingLyu BingqingLyu commented Apr 27, 2026

Copy link
Copy Markdown
Owner

Summary

Adds three new ways to attach images to the prompt and unifies them with the existing Cmd+V path under a single [Image #N] placeholder UX.

Closes QwenLM#3518

What's new

  1. Data URL text paste — paste data:image/<type>;base64,<payload> (devtools "Copy as data URL", chat messages containing embedded images, etc.).
  2. Raw base64 paste — accepted only when the decoded prefix matches a known image magic (PNG / JPEG / GIF / WebP / BMP / TIFF). JWTs, hashes, and base64-looking text without a valid image magic are rejected so normal paste flows aren't hijacked.
  3. Drag an image file into the terminal — works on terminals that wrap the drop in bracketed paste AND on terminals (macOS Terminal.app is the big one) that synthesize the drop as a rapid burst of individual keystrokes, via a simple < 10 ms keystroke-gap heuristic + a 150 ms debounced scan of buffer.text's trailing token.

Unified UX

All four sources — Cmd+V binary clipboard (existing), data URL, raw base64, drag-drop — converge on:

  • Inline [Image #N] placeholder inserted at the cursor.
  • Matching chip in the attachment row.
  • Monotonic counter per input, resets on submit.
  • [Image #N]@<relative path> substitution at submit time so the model sees each image exactly where the user placed it.
> please compare [Image #1] against [Image #2]
Attachments: [Image #1] [Image #2]

Attachments added without a placeholder (legacy code paths) keep rendering as [filename], so other callers are unchanged.

Code

  • clipboardUtils.ts:
    • tryDecodeBase64Image(text) — data URL + raw base64, magic-byte sniff.
    • saveDecodedImage(buf, ext, dir) — persists decoded bytes to the shared clipboard temp dir.
    • detectDraggedImagePath(text) — validates a token is an existing local image file; supports single/double-quoted paths and escaped spaces.
    • Existing clipboardHasImage / saveClipboardImage / cleanupOldClipboardImages unchanged.
  • InputPrompt.tsx:
    • Paste branch order: pasteImage → data URL / raw base64 → drag-drop paste path → large paste placeholder → small paste.
    • Keystroke-burst detection fallback for drag-drops that arrive as typing (see constants DRAG_BURST_MAX_INTERVAL_MS = 10, DRAG_MIN_BURST_CHARS = 4, DRAG_CHECK_DEBOUNCE_MS = 150).
    • [Image #N] allocator + [Image #N]@path substitution in handleSubmitAndClear.

Test plan

  • npm test --workspace packages/cli -- clipboardUtils InputPrompt143 pass, 1 Windows-skip, 0 fail
    • tryDecodeBase64Image: data URL, declared-vs-sniffed MIME (magic wins), raw base64 PNG, raw base64 JPEG, rejects text / JWT-like / empty / short / non-base64 data URL.
    • saveDecodedImage round-trips bytes to disk.
    • detectDraggedImagePath: existing file, quoted path, escaped-space path, non-image extension, missing file, directory, empty input, multi-line.
    • InputPrompt > base64 / data URL paste: single paste, sequential [Image #1]/[Image #2], fall-through to large-paste placeholder for non-images.
    • InputPrompt > drag-and-drop image paste: image path → [Image #1], non-image path → regular buffer handling.
  • npm run build --workspace packages/cli passes.
  • Manual on darwin-arm64, Node 22.17: data URL paste, screenshot Cmd+V, Finder drag onto Terminal.app, Finder drag onto iTerm, non-image drag, empty submit.

Backwards compatibility

  • All public signatures unchanged (clipboardHasImage, saveClipboardImage, cleanupOldClipboardImages).
  • Attachment chip rendering falls back to [filename] when no placeholder is set.
  • No change to multipart message assembly or to @path mentions outside the image paste flow.
  • No change to dependencies.

Notes

  • The burst heuristic thresholds are tuned to only trigger on mechanical input storms; human typing won't fire the detection.
  • The raw-base64 branch is conservative (decoded prefix must match a known image magic) so common base64-looking pastes — auth tokens, hashes, arbitrary data — are not turned into images.
  • Not in scope: the pre-existing @teddyzhu/clipboard binding issues on some platforms (Clipboard image paste (Cmd+V) silently fails on macOS — two root causes QwenLM/qwen-code#3517). This PR leaves that path untouched.

…age #N] placeholders

Adds three new ways to attach images to the prompt, all converging on a shared
[Image #N] placeholder UX inserted at the cursor:

- Paste a `data:image/...;base64,...` URL as text. Decoded after a
  conservative magic-byte check.
- Paste raw base64. Only accepted when the decoded prefix matches a known
  image magic (PNG/JPEG/GIF/WebP/BMP/TIFF) so JWTs and regular
  base64-looking text aren't hijacked.
- Drag an image file into the terminal. Supported both when the terminal
  wraps the drop in bracketed paste AND when it synthesizes the drop as a
  rapid burst of individual keystrokes (macOS Terminal.app), via a simple
  keystroke-burst heuristic (>= 4 chars with < 10 ms gaps) + a debounced
  scan of the buffer tail.

The existing Cmd+V clipboard path now also renders as [Image #N] for
consistency; attachments added without a placeholder fall back to the
legacy [filename] chip so other callers are unchanged.

At submit time, [Image #N] tokens in the message text are substituted with
@<relative path> so the model sees each image exactly where the user
placed it. Counter resets on submit.

Files
-----
- clipboardUtils.ts: add `tryDecodeBase64Image`, `saveDecodedImage`,
  `detectDraggedImagePath` (path is validated to exist, be a regular file,
  and have a recognized image extension). Accepts single-/double-quoted
  paths and escaped spaces as used by drag-drop.
- InputPrompt.tsx: unified paste branch (pasteImage -> base64 -> drag-drop
  paste -> large paste -> small paste), keystroke-burst detection for
  drag-drops that arrive as typing, [Image #N] allocator, attachment
  placeholder substitution on submit.
- Tests: 143/143 pass (1 Windows skip).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
it('accepts escaped spaces (terminal drag-drop style)', async () => {
const imagePath = path.join(tmp, 'a b.png');
await fs.writeFile(imagePath, PNG_MAGIC);
const escaped = imagePath.replace(/ /g, '\\ ');
Repository owner deleted a comment from github-actions Bot Apr 29, 2026
@BingqingLyu BingqingLyu added auto-merge-candidate auto-merge-candidate LOW risk, no cross-PR dependencies — safe to merge after quick review and removed conflicting-group-1 labels May 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-merge-candidate LOW risk, no cross-PR dependencies — safe to merge after quick review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow pasting images as base64 / data-URL text and dragging image files, with unified [Image #N] placeholders

3 participants