Skip to content

fix(core): preserve image_url content parts for DeepSeek vision model - #10026

Closed
nachoeverywhere wants to merge 1 commit into
QwenLM:mainfrom
nachoeverywhere:fix/deepseek-vision-image-parts
Closed

fix(core): preserve image_url content parts for DeepSeek vision model#10026
nachoeverywhere wants to merge 1 commit into
QwenLM:mainfrom
nachoeverywhere:fix/deepseek-vision-image-parts

Conversation

@nachoeverywhere

@nachoeverywhere nachoeverywhere commented Aug 25, 2026

Copy link
Copy Markdown

What this PR does

The DeepSeek provider was flattening every message into a plain string for all models, which turned image content into a [Unsupported content type: image_url] placeholder. When a user configures the vision model deepseek-v4-flash-vision-exp (with modalities.image), an image it reads via read_file never reaches the model — the request silently degrades to text. This PR stops flattening content when the config declares image support, so image_url parts are forwarded to the vision model as-is, while text-only models keep the existing placeholder behavior.

Why it's needed

DeepSeek's text models require message.content to be a plain string, which is why the provider flattens arrays and substitutes non-text parts. But the dedicated vision model accepts OpenAI content parts with image_url. Because flattening was unconditional, images read via read_file were replaced with [Unsupported content type: image_url] and the vision model never saw them, making vision-capable configs silently text-only. Users who set deepseek-v4-flash-vision-exp as their active model (a documented vision model) get a broken experience with no error.

Reviewer Test Plan

How to verify

Configure deepseek-v4-flash-vision-exp as the active model with modalities.image: true, then ask the agent to look at an image (or read_file a .png). Before this PR the request body contains [Unsupported content type: image_url] in place of the image; after this PR the image_url content part is preserved and sent to the vision model. The regression test asserts that image_url parts survive when modalities.image is set.

Evidence (Before & After)

N/A — non-UI change (internal request shaping, no user-facing difference). See the regression test for the before (placeholder) vs after (preserved image_url) of the request body.

Tested on

OS Status
🍏 macOS
🪟 Windows N/A
🐧 Linux N/A

Environment (optional)

Unit tests via npm run test --workspaces (core package, vitest run); 28 tests including the new regression case.

Risk & Scope

  • Main risk or tradeoff: vision-capable models switch from flattened-string to content-part messages. Only triggers when modalities.image === true, so text-only configs are unaffected.
  • Not validated / out of scope: a live round-trip against the DeepSeek vision API; the DeepSeek text behavior is covered by existing tests.
  • Breaking changes / migration notes: none — no config or CLI surface changed.

Linked Issues

Fixes #10027

中文说明

这个 PR 做了什么

DeepSeek 提供商此前将所有消息扁平化为纯字符串,这会把图片内容转换成 [Unsupported content type: image_url] 占位符。当用户配置视觉模型 deepseek-v4-flash-vision-exp(并设置 modalities.image)时,它通过 read_file 读取的图片永远不会到达模型——请求会静默退化为纯文本。此 PR 在配置声明支持图片时停止扁平化内容,从而将 image_url 部分原样转发给视觉模型,而纯文本模型则保留原有的占位符行为。

为什么需要

DeepSeek 的文本模型要求 message.content 是纯字符串,这也是提供商扁平化数组并用占位符替代非文本部分的原因。但专用的视觉模型接受带有 image_url 的 OpenAI 内容部分。由于扁平化是无条件的,通过 read_file 读取的图片会被替换为 [Unsupported content type: image_url],视觉模型从未真正看到它们,使得配置了视觉能力的模型静默地变成纯文本。用户将 deepseek-v4-flash-vision-exp 设为主模型(一个文档化的视觉模型)时会得到一个没有报错的破损体验。

Reviewer 测试计划

如何验证

deepseek-v4-flash-vision-exp 配置为主模型并设置 modalities.image: true,然后让智能体查看一张图片(或 read_file 一个 .png)。在此 PR 之前,请求体中图片位置是 [Unsupported content type: image_url];在此 PR 之后,image_url 内容部分被保留并发送给视觉模型。回归测试断言当 modalities.image 被设置时 image_url 部分能够存活。

前后证据(Before & After)

不适用——非 UI 变更(内部请求整形,无用户可见差异)。请参阅回归测试中的请求体前后对比(占位符 vs 保留的 image_url)。

测试环境

OS 状态
🍏 macOS
🪟 Windows N/A
🐧 Linux N/A

环境(可选)

通过 npm run test --workspaces(core 包,vitest run)运行单元测试;28 个测试,包括新增的回归用例。

风险与范围

  • 主要风险或权衡:视觉模型从扁平化字符串切换为内容部分组成。仅在 modalities.image === true 时触发,因此纯文本配置不受影响。
  • 未验证 / 超出范围:针对 DeepSeek 视觉 API 的在线往返;DeepSeek 文本行为由现有测试覆盖。
  • 破坏性变更 / 迁移说明:无——未改变任何配置或 CLI 表面。

关联 Issue

Fixes #10027

DeepSeek's text models require message content to be a plain string, but
the dedicated vision model (deepseek-v4-flash-vision-exp) accepts OpenAI
content parts with image_url. The DeepSeek provider always flattened
every non-text part into a `[Unsupported content type: image_url]`
placeholder, so images read via read_file never reached the vision model
and the request silently degraded to text.

Skip flattening when the config declares image support
(`modalities.image`), so image_url parts are sent as-is to the vision
model while text-only models keep the existing placeholder behavior.

Adds a regression test for the vision case.
@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Qwen Triage finishedview run. See the stage comments in this thread for the result.

Qwen Triage 已完成 —— 查看运行。结果见本线程中的各阶段评论。

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, @nachoeverywhere! Before we can review the change itself, the PR description needs to follow the repository's pull request template.

The current body uses free-form headings (Summary, Changes, Test plan, Screenshots / Demo), but none of the template's required sections are present:

  • ## What this PR does
  • ## Why it's needed
  • ## Reviewer Test Plan (with ### How to verify, ### Evidence (Before & After), and the ### Tested on OS table)
  • ## Risk & Scope
  • ## Linked Issues
  • the bilingual <details> Chinese summary the template asks for

Please edit the PR body to follow the template: https://github.com/QwenLM/qwen-code/blob/main/.github/pull_request_template.md

The substance you've already written maps cleanly onto those sections — this is a reformat, not new work. Once the body follows the template, re-trigger triage (@qwen-code /triage) and we'll continue with the review.

中文说明

感谢提交 PR,@nachoeverywhere!在进入代码审查之前,PR 描述需要按照仓库的 pull request 模板来填写。

当前正文使用的是自由格式的小节(SummaryChangesTest planScreenshots / Demo),模板要求的小节一个都没有:

  • ## What this PR does
  • ## Why it's needed
  • ## Reviewer Test Plan(包含 ### How to verify### Evidence (Before & After) 以及 ### Tested on 操作系统表格)
  • ## Risk & Scope
  • ## Linked Issues
  • 模板要求的双语 <details> 中文说明

请按照模板编辑 PR 正文:https://github.com/QwenLM/qwen-code/blob/main/.github/pull_request_template.md

你已经写好的内容可以很自然地对应到这些小节——这只是重新排版,不需要新增内容。正文符合模板后,重新触发 triage(@qwen-code /triage),我们会继续审查。

Qwen Code · qwen3.8-max

@nachoeverywhere

Copy link
Copy Markdown
Author

@qwen-code /triage

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Sandboxed verification: ⚠️ incomplete — cancelled - workflow run

The verification job was cancelled before producing a report.

中文 — 判定:⚠️ 未完成 · 已取消

验证作业在生成报告前被取消。

Qwen Code · sandboxed verification

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

tmux real-user testing: cancelled - workflow run

The testing job was cancelled before producing a verdict. See the workflow run for details.

Qwen Code · tmux real-user testing

@nachoeverywhere

Copy link
Copy Markdown
Author

Thanks for the review and for the related work in #9854 — it targets the same root cause with the same modalities.image gating, and the team's PR (tied to #9832) already covers both the official endpoint and custom gateways. I wasn't aware of the existing report/PR when I opened this; apologies for the overlap.

Closing this in favor of #9854, which is the right home for the change — earlier, maintained by the team, and with broader regression coverage. Happy to help with anything else if a piece of this is useful there; feel free to tag me.

@nachoeverywhere

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/self-reported The linked issue was opened by the PR author (self-reported)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DeepSeek vision model (deepseek-v4-flash-vision-exp) drops image_url content parts

2 participants