feat(cli): add /model --vision for a fallback vision model - #5778
Conversation
Add `/model --vision <id>` and an interactive vision-model picker to configure an image-capable model the vision bridge borrows when a text-only main model receives an image, mirroring the `/model --fast` pattern (flat `visionModel` setting + runtime `setVisionModel`). An explicit vision model takes precedence over same-provider auto-select and may cross providers; an unreachable pin falls back to auto-select rather than firing the bridge at a model it can't reach. Also fixes two vision-bridge issues surfaced while verifying #5126: - The gate read stale modalities after a runtime model switch, so an @image sent right after switching to a multimodal model could still trigger the bridge. Refresh model-derived modalities on the qwen-oauth hot-update paths (setModel default-model + handleModelChange) so the gate reflects the current model. - The transcription was appended after a now-empty "Content from <file>:" header, leading the primary model to re-read the image file and ignore the transcript. Stand the transcription in the image's slot and tell the model the image cannot be read by a tool. Closes #5597
|
Thanks for the PR! Template looks good ✓ On direction: this is the natural completion of voice dictation across all three surfaces — CLI had it, Web Shell landed it in #5755, desktop is the last gap. A user who configures voice in the CLI loses it the moment they switch to the desktop app, and desktop is exactly where hands-free input is most valuable (tablets, multi-monitor). Clearly aligned with the product's cross-surface parity goal. On approach: the scope matches the goal. Porting the CLI voice pipeline to Electron is inherently multi-file — renderer capture hooks, a loopback WS server, credential resolution, SSRF guard, batch + realtime transports, settings UI, i18n (7 locales). The two-process split (renderer captures mic → loopback WS → main process transcribes) is the right security model: provider credentials never reach the renderer. The acknowledged duplication with CLI voice code is acceptable — extracting a shared voice package is explicitly out of scope and the code is well-tested where it lives. Moving on to code review. 🔍 中文说明感谢贡献! 模板完整 ✓ 方向:这是语音听写在三个界面上的自然补全——CLI 已有,Web Shell 在 #5755 落地,桌面端是最后一个缺口。用户在 CLI 配置好语音后切到桌面端就失去了这个能力,而桌面端恰恰是解放双手的输入最有价值的场景(平板、多显示器)。与产品的跨界面一致性目标高度对齐。 方案:范围与目标匹配。把 CLI 语音管线移植到 Electron 天然涉及多个文件——renderer 采集 hook、loopback WS 服务端、凭证解析、SSRF 防护、批量+实时两种 transport、设置界面、i18n(7 种语言)。双进程分离(renderer 采集麦克风 → loopback WS → 主进程转写)是正确的安全模型:provider 凭证不会进入 renderer。与 CLI 语音代码的重复可以接受——抽取共享语音包明确在范围外,且代码在各自位置已有良好测试覆盖。 进入代码审查 🔍 — Qwen Code · qwen3.7-max |
There was a problem hiding this comment.
Pull request overview
Adds an explicit /model --vision <model-id> workflow (plus interactive picker) to pin the vision-bridge transcription model used when the primary model is text-only and receives images, while also addressing two vision-bridge correctness issues discovered during earlier verification (modality refresh on hot model switches; transcript placement to prevent redundant read_file).
Changes:
- Introduces a new
visionModelsetting (CLI + VS Code companion schema) and runtimesetVisionModel()path;/model --visionsupports interactive picking and non-interactive reporting. - Updates vision bridge output assembly to replace image parts in-place with the transcript/note and strengthens the “don’t re-read the image” framing.
- Fixes stale modality gating after qwen-oauth runtime
/modelswitches by refreshing model-derived modalities.
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/vscode-ide-companion/schemas/settings.schema.json | Adds visionModel setting schema entry for VS Code companion. |
| packages/core/src/services/visionBridge/vision-bridge-service.ts | Places transcript/note into the image slot via replaceImagesWithText; improves instruction framing. |
| packages/core/src/services/visionBridge/vision-bridge-service.test.ts | Adds regression test ensuring transcript is inserted in-place and trailing parts remain ordered. |
| packages/core/src/services/visionBridge/image-part-utils.ts | Introduces replaceImagesWithText helper for in-place replacement of image parts. |
| packages/core/src/services/visionBridge/image-part-utils.test.ts | Adds tests for replaceImagesWithText. |
| packages/core/src/models/modelsConfig.ts | Refreshes model-derived defaults on qwen-oauth hot switch to default model. |
| packages/core/src/models/modelsConfig.test.ts | Adds test ensuring modalities refresh on qwen-oauth hot switch. |
| packages/core/src/config/config.ts | Adds visionModel param + runtime setter, resolves explicit vision model selection, and hot-updates modalities in oauth paths. |
| packages/core/src/config/config.test.ts | Adds tests for explicit vs auto vision-bridge model selection behavior. |
| packages/cli/src/ui/hooks/useModelCommand.ts | Adds vision-model dialog mode state. |
| packages/cli/src/ui/hooks/slashCommandProcessor.ts | Routes vision-model dialog opening action. |
| packages/cli/src/ui/contexts/UIStateContext.tsx | Extends UI state to carry isVisionModelMode. |
| packages/cli/src/ui/contexts/UIActionsContext.tsx | Extends actions typing to support visionModelMode. |
| packages/cli/src/ui/components/ModelDialog.tsx | Adds “Select Vision Model” mode; persists visionModel and syncs runtime config. |
| packages/cli/src/ui/components/ModelDialog.test.tsx | Tests storing authType-qualified selectors in vision mode without switching main model. |
| packages/cli/src/ui/components/DialogManager.tsx | Wires vision mode into dialog manager and settings shortcut. |
| packages/cli/src/ui/commands/types.ts | Adds vision-model to dialog action union. |
| packages/cli/src/ui/commands/modelCommand.ts | Implements /model --vision flag behavior, completion, and non-interactive reporting. |
| packages/cli/src/ui/commands/modelCommand.test.ts | Adds tests for /model --vision set + reject flows. |
| packages/cli/src/ui/AppContainer.tsx | Plumbs vision mode dialog state through UI providers. |
| packages/cli/src/config/settingsSchema.ts | Adds CLI settings dialog entry for visionModel. |
| packages/cli/src/config/config.ts | Wires CLI settings visionModel into core Config construction. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Code Review (re-run on latest commits)Independent proposal before reading the diff: add a The PR's implementation matches this proposal and the latest two commits (
No critical blockers found. No AGENTS.md violations. Unit TestsAll 673 tests pass across affected suites: Real-Scenario Testing (tmux)Built the bundle from the PR branch ( All four paths behave correctly:
中文说明代码审查(最新 commit 重新审查)独立提案与 PR 实现匹配。最新两个 commit 解决了之前 CHANGES_REQUESTED 的两个关键发现:
未发现关键阻断问题,无 AGENTS.md 违规。 单元测试全部 673 个测试通过。 真实场景测试 (tmux)构建 bundle 后测试四条命令路径,行为均正确。特别是主模型拒绝路径验证了前次审查的关键修复。 — Qwen Code · qwen3.7-max |
|
This PR has come a long way since the initial review. The author pushed four additional commits addressing the CHANGES_REQUESTED feedback, and the result is a solid, well-hardened feature. Stepping back: the problem is real — text-only models silently dropping images is one of the more frustrating user experiences. The The latest commits add real substance beyond the initial implementation: the My independent proposal matched the PR's approach, and the PR went further in good ways. The All 673 unit tests pass. The tmux testing verified all four command paths including the new primary-model rejection. The one lingering issue — LGTM, approving. ✅ 中文说明这个 PR 自初始审查以来有了很大改进。作者推送了四个额外 commit 解决了 CHANGES_REQUESTED 反馈,结果是一个扎实、加固良好的功能。 问题是真实的——纯文本模型静默丢弃图片是令人沮丧的用户体验之一。 最新 commit 增加了实质性内容: LGTM,批准。✅ — Qwen Code · qwen3.7-max |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Overall the PR is well-structured — follows the established --fast / --voice patterns cleanly. Three suggestions below on model resolution robustness, the dialog filter gap, and debug logging.
— qwen3.7-max via Qwen Code /review
…ity refresh Self-review follow-ups on the /model --vision work: - Warn (but still honor) when /model --vision or the picker pins a model that isn't image-capable, reusing the vision bridge's isImageCapable so the explicit and auto-select paths agree on what counts as vision-capable. isImageCapable is now exported for the CLI to share. - Exclude fastOnly models from --vision validation so it matches the completion filter. - Add a Config-level test asserting handleModelChange refreshes modalities on the qwen-oauth hot path (the vision-bridge gate fix that lacked direct coverage).
Add zh/zh-TW/en entries for the /model --vision picker warning shown when a pinned model isn't image-capable, matching the existing vision-string translations.
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Overall the PR is well-structured and the vision bridge refactor (replaceImagesWithText standing the transcript in the image's slot) is clean. Tests for the core logic pass (435 tests across 6 suites). Three smaller findings below — two latent correctness issues and one i18n inconsistency.
Build/typecheck failures on this branch are pre-existing in packages/cli/src/ui/components/BaseTextInput.tsx (from PR #5772) and unrelated to this PR.
— qwen3.7-max via Qwen Code /review
The vision bridge printed the full transcription in its notice AND fed it to the primary model, which then re-stated it — so the user read the same image description twice (once in the notice box, once in the answer). - The notice now shows only the egress-disclosure header; the transcription is fed to the primary model and surfaced in its answer, so it is no longer duplicated. Removes the now-unused transcript truncation / control-char stripping helpers. - The bridge model is instructed to transcribe/describe, not answer the user request (the user's intent is carried as a focus hint, not a question), so its output is context for the primary model rather than a second competing answer.
- Register 'Vision Model' / 'Select Vision Model' in en/zh/zh-TW so the vision picker and confirmation aren't English-only for non-English users (mirrors the existing Voice Model entries). - formatNonVisionModelWarning reuses the dialog's already-registered t() key instead of a plain English literal, keeping both paths i18n-consistent. - Add the missing bare '/model --vision' tests (interactive dialog + non-interactive current-model report), mirroring the --voice coverage.
The inline transcript display was dropped in 14de333 (it duplicated the description already fed to the primary model), leaving `transcript` set on the result but never read in any production path. Remove the field and its now-redundant test — the untrusted-wrapper assertion it carried is already covered by 'converts images to an untrusted text block on success'.
|
@qwen-code /triage |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
— qwen3.7-max via Qwen Code /review
wenshao
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
— qwen3.7-max via Qwen Code /review
Address review feedback on #5778: - config: provider-aware isCurrentPrimaryModel() so a cross-provider namesake (e.g. anthropic:shared-model vs an openai shared-model primary) stays an eligible vision pin; resolveVisionModelSelection now also filters fast/voice- only models (a settings.json pin bypasses the slash command) and warns on every silent-drop path so a stale pin is debuggable. - modelCommand/ModelDialog: reject pinning the current primary as the vision model at set time (ignored at runtime) instead of persisting a dead pin. - useModelCommand: vision mode suppresses voice so the dialog can't end up in two specialized modes at once. - HistoryItemDisplay: vision_notice gets 0 top margin like its notice peers. - vision-bridge-service: log transcription metadata only (model + length), never the raw text — screenshots carry tokens/PII and debug logs can be shared in support bundles. PR #5778.
…ings, tests Address the remaining #5778 review feedback: - baseLlmClient/sideQuery/vision-bridge: add an opt-in `failClosed` to the per-model resolve path; the vision bridge sets it so a missing cross-provider credential fails the conversion instead of silently routing image payloads at the text-only main generator (which BaseLlmClient otherwise falls back to). - ModelDialog: reject a vision pin that maps to multiple same-provider endpoints (the persisted authType:modelId can't disambiguate base URLs), mirroring the voice-mode duplicate guard. - modelCommand: wrap the non-interactive vision status message in t() with en/zh/zh-TW locale entries. - SettingsDialog: wire the Vision Model row into the sub-dialog dispatch so Enter / Right-Arrow open the picker (like fastModel). - image-part-utils: document the multi-image drop / "don't read_file" contract. - tests: fail-closed resolve paths, primary-guard fallback, vision initialIndex, authType-qualified + malformed vision selectors, failure-path image removal. PR #5778.
|
@qwen-code /triage |
✅ Local verification report (maintainer)Built and ran this PR locally as a merge reference. Beyond running the suites I mutation-tested the three load-bearing changes — the explicit-pin resolution order, and both folded-in fixes — by breaking each guard in the production code and confirming the matching test fails, then reverting. All green; the new tests genuinely catch the regressions they target. Environment
Automated tests — all green (972 passed)
Mutation testing — the new tests are load-bearingFor each, I broke the production guard and confirmed the test goes red, then reverted (production
Design review (read in full)
Scope noteThe live bridge firing against real vision models (dashscope) was not independently re-run here — it needs live provider credentials and is evidenced by the author's real-model screenshots. The command/picker/warning UX and the routing logic are covered by the unit suites above, three of which I confirmed load-bearing by mutation. CI
VerdictCareful, conservative design (fail-safe resolution, provider-aware primary exclusion, privacy-aware logging), additive and default-off (empty 中文版(完整对应)✅ 本地验证报告(维护者)在本地构建并运行了本 PR 作为合并参考。除了跑测试套件,我还对三处 load-bearing 改动做了变异测试——显式 pin 的解析顺序,以及两个折叠进来的修复——把生产代码里对应的守卫逐个改坏、确认对应测试失败,再还原。全绿;新测试确实能捕捉它们针对的回归。 环境
自动化测试 —— 全绿(972 通过)
变异测试 —— 新测试是 load-bearing每一项我都把生产守卫改坏、确认测试变红,再还原(之后生产
设计审查(完整通读)
范围说明针对真实视觉模型(dashscope)的桥实际触发这次没有独立重跑 —— 它需要真实 provider 凭证,由作者的真实模型截图佐证。命令/选择框/警告 UX 与路由逻辑由上述单测覆盖,其中三处我已用变异确认 load-bearing。 CI
结论设计谨慎保守(失败向安全侧回退、provider 感知地排除 primary、隐私感知的日志),additive 且默认关闭( |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Code Review Summary
This review covers commit 5a452216 (latest in the PR).
- Findings: 3 (0 Critical, 3 Suggestions)
- Verdict: COMMENT
- Typecheck: 4 errors from rebase-needed stale test files — already discussed by @wenshao on 2026-06-25, excluded from this review.
- Low-confidence: 3 additional items flagged for human review (failure notice hides error reason, skipped+egress dim rendering, auto-select vs explicit pin identity semantics).
| # | Severity | File | Line | Issue |
|---|---|---|---|---|
| 1 | Suggestion | config.test.ts |
415 | Missing unit tests for 3 vision-model runtime guards |
| 2 | Suggestion | modelCommand.ts |
515 | Missing i18n keys for primary-model rejection messages |
| 3 | Suggestion | ModelDialog.tsx |
593 | Missing test for vision-mode primary-model rejection |
🤖 Review Method
- Tools: eslint (0 issues), tsc (4 rebase-needed errors, excluded)
- Review agents: 6 parallel LLM reviewers + 1 verification agent
- Reverse audit: 2 rounds (converged)
- Context: 76 prior inline comments analyzed, 0 overlap with current findings
| expect(config.getSystemPrompt()).toBeUndefined(); | ||
| }); | ||
|
|
||
| describe('getDefaultVisionBridgeModel', () => { |
There was a problem hiding this comment.
[Suggestion] Missing unit tests for three vision-model runtime guards in config.ts.
The isCurrentPrimaryModel method has 5 branches (authType match, baseUrl match, bare-id match, all three differ, bare-id short-circuit) but no direct unit tests. Similarly, the fastOnly/voiceOnly early-return guards and the primary-itself rejection guard in resolveVisionModelSelection are only exercised indirectly through getDefaultVisionBridgeModel tests.
Consider adding direct tests:
describe('isCurrentPrimaryModel', () => {
it('returns true when authType and baseUrl match', () => { /* ... */ });
it('returns false when only authType matches', () => { /* ... */ });
it('falls back to bare id comparison', () => { /* ... */ });
});
describe('resolveVisionModelSelection guards', () => {
it('returns undefined when fastOnly is true', () => { /* ... */ });
it('returns undefined when voiceOnly is true', () => { /* ... */ });
it('returns undefined when explicit pin matches primary model', () => { /* ... */ });
});| type: 'message', | ||
| messageType: 'error', | ||
| content: t( | ||
| "'{{model}}' is the current primary model and cannot be used as the vision bridge. Choose a different image-capable model.", |
There was a problem hiding this comment.
[Suggestion] Missing i18n keys for primary-model rejection messages.
The t("⚠ '{{model}}' is the current primary model...") calls at this line (and the equivalent in ModelDialog.tsx:593) are not registered in any locale file (en.js, zh.js, zh-TW.js). While English works (key = value), Chinese users will see the raw English key with unfilled {{model}} placeholders.
Add these keys to all locale files, matching the pattern used for the vision-model unavailable messages already registered in en.js.
| ) { | ||
| setErrorMessage( | ||
| t( | ||
| "'{{model}}' is the current primary model and cannot be used as the vision bridge.", |
There was a problem hiding this comment.
[Suggestion] Missing test for the vision-mode primary-model rejection.
The isCurrentPrimaryModel branch here shows a rejection message and returns early, but there's no test for this path in ModelDialog.test.tsx. The existing vision-mode tests cover the success path and the non-image-capable warning, but not the primary-model rejection case.
it('rejects primary model in vision mode', async () => {
// set up vision model mode with same model as primary
// assert rejection message is rendered
});
What this PR does
Adds
/model --vision <model-id>(plus an interactive picker when run with no argument) to configure an image-capable model the vision bridge borrows when a text-only main model receives an image. It mirrors the existing/model --fastUX: a flatvisionModelsetting, a runtimesetVisionModel, completion, and a settings entry.Resolution order for the bridge model:
visionModelset and configuredvisionModelset but not configured (stale/typo)visionModelunsetThis PR also folds in two vision-bridge fixes that surfaced while verifying #5126 (both tracked in the issue thread):
/modelswitch. Switching to a multimodal model and immediately sending an@imagecould still trigger the bridge, because the gate read the previous model's modalities. The qwen-oauth hot-update paths (setModeldefault-model branch andhandleModelChange) now refresh model-derived modalities, so the gate reflects the current model.read_file/ ignored transcript. The transcription was appended after a now-emptyContent from <file>:header, so the primary model saw an empty header and re-read the image file instead of using the transcript. The transcription now stands in the image's slot, and the framing tells the model the image can't be read by a tool.Why it's needed
Text-only models (Qwen3.7-max, DeepSeek-V4-Pro, …) are a common main-model choice for coding quality, but sending a screenshot then silently drops the image or fails.
/model --visionlets you keep your preferred text model and still get image understanding transparently, without manually switching models back and forth. This is the explicit-config counterpart to the same-provider auto-select already in the bridge.Reviewer Test Plan
How to verify
Main model text-only (
qwen3.7-max), a vision model on the same endpoint (qwen3.7-plus):/model --vision qwen3.7-plus→ printsVision Model: qwen3.7-plus./model --visionwith no arg opens a "Select Vision Model" picker (defaults the highlight to the configured model).@some-image.png <question>→ bridge fires via the configured model, transcribes the image, and the text-only main model answers from the transcript. Only oneRead File(the@-resolution), no redundant re-read./model qwen3.7-plus(switch main model to multimodal), then@some-image.png …→ no bridge notice; the image goes straight to the multimodal model.Unit:
npm run testfor the touched suites (config, modelsConfig, vision-bridge, modelCommand, ModelDialog).npm run typecheck && npm run lint.Evidence
Full case regression on the latest branch head, real dashscope models, color tmux capture. Main text-only model
qwen3.7-max; image-capable modelsqwen3.7-plus/qwen3-vl-plus.The bridge fires only when the main model is text-only — a multimodal main keeps the image:
Text-only main → routes to the configured
visionModel. HerevisionModel = qwen3-vl-plus(deliberately a different model from the multimodal one below); the image is transcribed via that model before the text-only main answers.Multimodal main → bridge stays off. The same
visionModelis still configured, but with a multimodal main (qwen3.7-plus) the image goes straight to it — noConvertednotice, not routed to the vision model.visionModelunset → same-provider auto-pick. With no explicit pin, a text-only main borrows a same-provider image-capable model./model --visionUX/model --vision <id>sets and confirms the vision model:/model --visionwith no argument opens the picker:/model --vision <text-only-model>warns it is not image-capable (and still falls back to auto-pick on images):Tested on
Environment (optional)
Built CLI (
packages/cli/dist/index.js) against live dashscope models.Risk & Scope
visionModelis allowed to cross providers (per the issue), so a deliberately-pinned model sends the image to that provider. Auto-select stays same-provider only; an unreachable pin falls back to auto-select rather than firing blindly.Linked Issues
Closes #5597
中文说明
这个 PR 做了什么
新增
/model --vision <model-id>(不带参数时打开交互式选择框),用于配置一个具备图像能力的模型,作为「视觉桥」:当纯文本主模型收到图片时,借这个模型把图转成文字。整体对标现有/model --fast:扁平visionModel设置、运行时setVisionModel、补全、设置项。桥模型的解析顺序:
visionModel已设且已配置visionModel已设但未配置(陈旧/拼错)visionModel未设本 PR 同时合入两个在验证 #5126 时发现的视觉桥修复(issue 里有记录):
/model切换后 gate 读到陈旧 modality。 切到多模态模型后紧接着发@图仍可能误触发桥,因为 gate 读的是上一个模型的 modalities。现在 qwen-oauth 热更新路径(setModel默认模型分支和handleModelChange)会刷新模型派生的 modalities,让 gate 反映当前模型。read_file/ 忽略转写。 转写之前被追加在一个已经空了的Content from <file>:头之后,主模型看到空头就去重新读图、而不用转写。现在转写就地放在图片原来的位置,并明确告诉模型图片无法用工具读取。为什么需要
纯文本模型(Qwen3.7-max、DeepSeek-V4-Pro 等)常被选作主模型(看中编码质量),但发截图会被静默丢弃或报错。
/model --vision让你保留偏好的文本模型,同时透明地获得图像理解,无需来回手动切模型。这是桥已有的「同-provider 自动挑选」的显式配置版。复核测试计划
主模型纯文本(
qwen3.7-max),同端点视觉模型(qwen3.7-plus):/model --vision qwen3.7-plus→ 打印Vision Model: qwen3.7-plus;不带参打开「Select Vision Model」选择框(默认高亮到已配置的模型)。@某图.png <问题>→ 桥经配置的模型触发、转写、纯文本主模型用转写作答;只有一次Read File(@解析),无重复读。/model qwen3.7-plus(切到多模态主模型),再@某图.png …→ 无桥提示,图直发多模态模型。单测:上述受影响 suite +
npm run typecheck && npm run lint。风险与范围
visionModel允许跨 provider(按 issue 要求),所以刻意钉的模型会把图发到那个 provider。自动挑选仍只限同-provider;够不到的钉值回退到自动挑选而非盲触发。Closes #5597