Skip to content

[Bugfix][Rust Frontend][Multimodal] Align DeepSeek V4.1 and Kimi K3 media with rendered placeholders - #56366

Merged
BugenZhao merged 4 commits into
vllm-project:mainfrom
reidliu41:fix-rust-multimodal-render-order
Sep 14, 2026
Merged

BugenZhao merged 4 commits into
vllm-project:mainfrom
reidliu41:fix-rust-multimodal-render-order

Conversation

@reidliu41

@reidliu41 reidliu41 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Purpose

Rust chat rendering and multimodal extraction previously determined media
order independently.

DeepSeek V4.1 and Kimi K3 can reorder tool results to match the preceding
assistant tool-call order. The rendered image placeholders therefore followed
tool-call order, while media extraction still followed raw request order. When
tool results arrived out of order, images could be silently associated with
the wrong tool results.

DeepSeek V4.1 can also omit historical developer messages. Images from those
messages were still extracted even though no corresponding placeholder was
rendered, causing multimodal placeholder expansion to fail.

Make renderers report media source locations while emitting placeholders, and
use that order during multimodal extraction. DeepSeek V4.1 and Kimi K3 provide
their exact rendered media order. Renderers that preserve request order retain
the existing behavior through the default implementation.

Test Plan

  cargo nextest run --locked -p vllm-chat --lib \
    -E 'test(media_order_follows_reordered_tool_results) | test(media_order_excludes_dropped_historical_developer_content) |
    test(extract_media_parts_follows_renderer_order)'

  cargo nextest run --locked -p vllm-chat

  cargo nextest run --locked -p vllm-server --lib  -E 'test(non_stream_chat_image_url_reaches_engine_mm_features)'

  cargo fmt  --all -- --check

  cargo clippy --workspace --all-targets --all-features --locked -- -D warnings

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Signed-off-by: reidliu41 <reid201711@gmail.com>
@mergify mergify Bot added deepseek Related to DeepSeek models rust kimi k3 DSv4 bug Something isn't working labels Sep 11, 2026
@reidliu41
reidliu41 marked this pull request as ready for review September 11, 2026 01:09

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added the DSv4.1 Related to DeepSeek-V4.1 models label Sep 12, 2026
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>

@BugenZhao BugenZhao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @reidliu41 this is a good catch! I've added a commit to put media_order into RenderedPrompt and use None for default ordering.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@github-project-automation github-project-automation Bot moved this from To Triage to Ready in gpt-oss Issues & Enhancements Sep 14, 2026
@BugenZhao BugenZhao added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 14, 2026
@BugenZhao

Copy link
Copy Markdown
Member

/ci run

@BugenZhao
BugenZhao enabled auto-merge (squash) September 14, 2026 09:55
@github-actions

Copy link
Copy Markdown

@reidliu41, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #88787 for commit 56440a17b610.

@BugenZhao
BugenZhao merged commit 7b1ea3f into vllm-project:main Sep 14, 2026
35 of 37 checks passed
Shreya-gaur pushed a commit to Shreya-gaur/vllm_private that referenced this pull request Sep 14, 2026
…edia with rendered placeholders (vllm-project#56366)

Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
ItsRoy69 pushed a commit to ItsRoy69/vllm that referenced this pull request Sep 15, 2026
…edia with rendered placeholders (vllm-project#56366)

Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working deepseek Related to DeepSeek models DSv4.1 Related to DeepSeek-V4.1 models gpt-oss Related to GPT-OSS models k3 kimi ready ONLY add when PR is ready to merge/full CI is needed rust

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants