fix(core): recognize new DeepSeek/GLM vision models in modality auto-detection - #10278
Conversation
…detection Add targeted entries to MODALITY_PATTERNS so name-based modality detection grants image input to the vision models reported in #10270: - deepseek-v4-flash-vision-exp (and any DeepSeek -vision variant) previously matched the family-wide [/^deepseek/, {}] text-only entry. - glm-4.6v / glm-5v-turbo belong to the same v-suffix vision family as the already-recognized glm-4.5v; generalize that entry to [/^glm-[0-9.]+v/, { image: true }]. - glm-5.3-flash natively integrates vision input but has no v suffix, so it gets its own precise entry ahead of the text-only GLM patterns. Without these entries the models fall through to the text-only default, so createMediaContentPart() silently replaces attached images with a text placeholder and isImageCapable() excludes the models from vision-bridge candidates. Explicit modalities overrides were already honored and are untouched. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
|
✅ Qwen Triage finished — CI landed green on ✅ Qwen Triage 已完成 —— |
|
Thanks for the PR! Template looks good ✓ Problem: observed bug, well evidenced. Linked issue #10270 reproduces it on v0.22.2 against the real DeepSeek and Zhipu endpoints — images are silently replaced by the text placeholder, with screenshots and a control test showing the explicit Direction: aligned. Size: core paths touched ( Approach: scope feels right. Targeted entries ahead of the family catch-alls, following the existing first-match-wins convention and the Doubao Risk: no elevated risk signals — no changed file matches the revert-correlated high-risk paths. Moving on to code review. 🔍 中文说明感谢贡献! 模板完整 ✓ 问题:已观测到的 bug,证据充分。关联 issue #10270 在 v0.22.2 上用真实的 DeepSeek 与智谱端点复现——图片被静默替换为文本占位符,附有截图,且对照实验证明显式设置 方向:对齐。 规模:触及核心路径( 方案:范围合理。在家族兜底模式之前加入精确条目,遵循现有"按序首次匹配"惯例以及表中已有的 Doubao 风险:无升级风险信号——改动文件均未命中与 revert 相关的高风险路径。 进入代码审查 🔍 — Qwen Code · qwen3.8-max Reviewed at |
Code reviewBefore reading the diff, my independent take on "four vision models fall through to the text-only default" was: add targeted vision entries to
No blockers, no convention issues — issue-tagged comments follow the precedent the Doubao section already sets, and the change is data-only. TestingCI on the reviewed commit, fetched via API (per gate rules, PR code is never executed here): Final CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行(取最新一次运行),省略 skipped,失败项排在最前。 No failures so far; the ubuntu unit suite is still running and the OS-matrix and integration checks are gated behind it. I won't poll — the finalize workflow updates this table once CI settles. Two attribution notes: the pre-patch red baseline ( Sandboxed verification would settle the remaining gap if desired: 中文说明代码审查:在读 diff 之前,我的独立思路就是"在家族兜底模式之前加入精确的视觉条目(按序首次匹配),并补充回归测试确保纯文本兄弟模型不受影响"——PR 正是这么做的,且经得起推敲:模式顺序正确( 测试:以上为被审提交在 CI 上的真实检查结果(门禁规则要求不得在此执行 PR 代码)。目前无失败:ubuntu 单元测试套件仍在运行,macOS/Windows 测试与集成测试在其身后排队;不做轮询,CI 结束后由 finalize 工作流更新表格。两点归属说明:未打补丁时的红色基线( — Qwen Code · qwen3.8-max Reviewed at |
|
Confidence: 4/5 — clean, minimal fix for a well-evidenced bug; the only soft spot is that the red baseline and vendor-endpoint behavior rest on the author's and reporter's numbers, which CI structurally cannot re-run. This is what a good detection-table fix looks like. The problem is real and thoroughly documented — silent image loss on four current vision models, reproduced on actual DeepSeek and Zhipu endpoints with a control test that pinpoints the detection layer. The fix is exactly the minimal change: 11 production lines of table entries placed correctly for first-match-wins ordering, with the subtler edges (the normalizer stripping I'd normally approve on the spot. The ubuntu unit suite is still running on this commit, so approval is deferred until CI lands green on 中文说明这是一次教科书式的检测表修复。问题真实且记录完备——四个现役视觉模型被静默丢图,报告者在 DeepSeek 与智谱真实端点复现,并用对照实验精确定位到检测层。修复恰好是最小改动:11 行生产代码的表条目,按"首次匹配生效"的顺序放置正确,较隐蔽的边界(规范化器剥离 — Qwen Code · qwen3.8-max Reviewed at |
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
doudouOUC
left a comment
There was a problem hiding this comment.
Reviewed exact head c363f65. No blocking issues found. I verified the ordered first-match patterns and normalization behavior, and traced every downstream consumer: modelConfigResolver, modelRegistry, ModelsConfig, and the vision bridge fallback. The four reported model IDs become image-capable while existing DeepSeek and GLM text-only siblings remain unchanged, and explicit modality overrides still take precedence. Official vendor documentation confirms the affected model capabilities. Locally, modalityDefaults and vision-bridge tests passed (123 tests total), and the core package typecheck passed. The repository-wide build reached the CLI package but is blocked in this worktree by the existing Ink selection type mismatch, outside this two-file core diff; the relevant remote unit, security, and desktop checks are green.
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
|
Released in v0.22.3. |
…-max Resolve five conflicts; main had superseded most of this branch: - modalityDefaults.ts (+ its test): keep main's newer classification — qwen3.8-max is image-only there (QwenLM#10195/QwenLM#10278), not image+video. - alibaba-token-plan.ts: keep this branch's deepseek-v4-flash entry (the part main never landed) and drop its duplicate qwen3.8-max record; main's later QwenLM#9383 entry, with thinkingMandatory, is authoritative. The matching branch assertion is dropped with it. - vscode-ide-companion subscriptionPlanDefinitions.{ts,test.ts}: take main's version, which now derives the plan list from the shared preset instead of the hand-kept copy this branch was updating. core: alibaba-token-plan + modalityDefaults 62 passed; vscode-ide-companion subscriptionPlanDefinitions 3 passed.
What this PR does
Extends the name-based modality auto-detection table (
MODALITY_PATTERNSinpackages/core/src/core/modalityDefaults.ts) so four newly reported vision models are detected as image-capable:deepseek-v4-flash-vision-exp(matched by a targeted entry for DeepSeek-visionvariants placed ahead of the family-wide text-only entry),glm-4.6vandglm-5v-turbo(the existingglm-4.5v-specific entry is generalized to av-suffix family pattern, keepingglm-4.5vcovered), andglm-5.3-flash(its own precise entry, since it integrates vision natively despite having novsuffix). Everything else in the table is unchanged.Why it's needed
On v0.22.2 and current main, these vision models fall through to the text-only default, so when a user attaches an image via a custom OpenAI-compatible provider without manually overriding
model.generationConfig.modalities,createMediaContentPart()silently replaces the image with a text placeholder before the request is sent and the model never sees the image. The same table feedsisImageCapable()in the vision bridge, so these models are also excluded from vision-bridge candidates. There is no user-visible warning, only a debug log line.Reviewer Test Plan
How to verify
Red/green unit-test repro against
defaultModalities():defaultModalities('<model>').image === truefordeepseek-v4-flash-vision-exp,glm-4.6v,glm-5v-turbo,glm-5.3-flash: 4 failed | 54 passed (exactly the four reported ids fail).cd packages/core && npx vitest run src/core/modalityDefaults.test.ts→ 58 passed (58). The four models now detectimage: true, and the regression assertions hold:deepseek-chat/deepseek-reasoner/ non-visiondeepseek-v4-flashstay text-only,glm-5/glm-4.7/glm-4.6(novsuffix) stay text-only, and existingglm-4.5v/ Qwen / Gemini entries are unchanged.npm run typecheck -w @qwen-code/qwen-code-core— clean.vision-bridge-service.test.tsalso run (65 passed) becauseisImageCapable()consumes this table.Evidence (Before & After)
N/A (non-UI detection-table change; test output above is the evidence).
Tested on
Environment (optional)
Unit tests only (
vitest); no vendor API keys needed.Risk & Scope
v-suffix pattern is generalized from^glm-4\.5vto^glm-[0-9.]+v— a strict superset that still matchesglm-4.5v(covered by the existing test) and tracks Zhipu'sv-suffix vision naming convention; the DeepSeek-visionmatch follows the repo precedent for explicit vision markers (see the Doubao section). Text-only DeepSeek/GLM models are protected by regression tests.modalitiesoverrides keep priority and are untouched.Linked Issues
Fixes #10270
中文说明
这个 PR 做了什么
扩展基于模型名的模态自动检测表(
packages/core/src/core/modalityDefaults.ts的MODALITY_PATTERNS),使四个新报告的视觉模型被识别为支持图片输入:deepseek-v4-flash-vision-exp(新增针对 DeepSeek-vision变体的精确条目,置于整个家族纯文本条目之前);glm-4.6v与glm-5v-turbo(把原先仅针对glm-4.5v的条目泛化为v后缀家族模式,glm-4.5v仍被覆盖);glm-5.3-flash(原生集成视觉但没有v后缀,单独给精确条目)。表中其余内容不变。为什么需要
在 v0.22.2 和当前 main 上,这些视觉模型落入纯文本默认值:用户通过自定义 OpenAI-compatible provider 附加图片且未手动覆盖
model.generationConfig.modalities时,createMediaContentPart()会在请求发出前把图片静默替换为文本占位符,模型永远看不到图片。同一张表也被 vision bridge 的isImageCapable()消费,这些模型因此被排除在 vision-bridge 候选之外。全程只有 debug 日志,没有用户可见警告。评审验证计划
如何验证
对
defaultModalities()做红/绿单元测试复现:deepseek-v4-flash-vision-exp、glm-4.6v、glm-5v-turbo、glm-5.3-flash的defaultModalities('<model>').image === true:4 failed | 54 passed(恰好是报告的四个模型名失败)。cd packages/core && npx vitest run src/core/modalityDefaults.test.ts→ 58 passed (58)。四个模型现在检出image: true,且回归断言成立:deepseek-chat/deepseek-reasoner/ 非 vision 的deepseek-v4-flash仍为纯文本;glm-5/glm-4.7/glm-4.6(无v后缀)仍为纯文本;既有glm-4.5v/ Qwen / Gemini 条目行为不变。npm run typecheck -w @qwen-code/qwen-code-core—— 通过。另跑了vision-bridge-service.test.ts(65 通过),因为isImageCapable()消费这张表。证据(修复前后)
N/A(非 UI 的检测表改动;以上述测试输出为证据)。
测试环境
环境(可选)
仅单元测试(
vitest),不需要厂商 API key。风险与范围
v后缀模式从^glm-4\.5v泛化为^glm-[0-9.]+v—— 严格超集,仍匹配glm-4.5v(既有测试覆盖),并遵循智谱的v后缀视觉命名惯例;DeepSeek-vision匹配沿用仓库中对显式视觉标记的先例(见 Doubao 段)。纯文本的 DeepSeek/GLM 模型有回归测试保护。modalities覆盖优先级不变,未被改动。关联 Issue
Fixes #10270