Skip to content

[Rust] multimodal generation back-ported on vllm v0.28.0 - #56421

Draft
biswapanda wants to merge 3 commits into
vllm-project:releases/v0.28.0from
biswapanda:feat/dynamo-vlm-v028-minimal-20260911
Draft

biswapanda wants to merge 3 commits into
vllm-project:releases/v0.28.0from
biswapanda:feat/dynamo-vlm-v028-minimal-20260911

Conversation

@biswapanda

@biswapanda biswapanda commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Add the minimal vLLM 0.28 Rust frontend support needed for Dynamo-backed Qwen3.5 image generation during RL training.

This is not a duplicate of #55047: that PR targets current main and includes broader multimodal work. This branch is based directly on releases/v0.28.0 and keeps only the v0.28 compatibility path required by the tested Dynamo stack.

Changes

  • Preserve native Generate sampling semantics across gRPC, including omitted temperature and explicit zero or false values.
  • Advertise separate capabilities for native sampling JSON and preprocessed multimodal features.
  • Accept image-only engine-ready multimodal features and forward them to TextRequest.mm_features.
  • Validate MessagePack structure, inline tensor layout, feature ranges, prompt bounds, and mixed raw/preprocessed inputs.
  • Bind cache identity to validated feature content instead of trusting caller-provided identifiers or hashes.
  • Report the engine world size required by Dynamo managed-RL discovery.
  • Accept both the older short engine-output tuple and vLLM 0.28's current 18-field tuple while discarding unused speculative-decode and routed-expert metadata without allocation.

The change intentionally excludes LoRA handling, audio/video support, chat model-limit refactors, and unrelated code from #55047.

Validation

  • cargo fmt --all -- --check
  • cargo check -p vllm-server --tests
  • cargo test -p vllm-server grpc::convert::tests:: — 22 passed
  • cargo test -p vllm-server preprocessed_image — 7 passed
  • cargo test -p vllm-server control_reports_server_and_model_info — 1 passed
  • cargo test -p vllm-engine-core-client --lib — 97 passed, including both output tuple lengths
  • Cross-language MessagePack fixture produced by Dynamo decodes successfully.
  • Five-step Qwen3.5-4B multimodal RL run passed on bis-rl-3 with Prime-RL main plus [Core] [Bugfix] Refactor block manager subsystem for better testability #3492, Dynamo [Installation]: #14398 plus the minimal image-only sidecar layer, and this PR: 5/5 orchestrator steps, 5/5 trainer steps, 2/2 trainable rollouts per step, zero rollout errors/cancellations, six NCCL weight receives, and successful pre/post image probes.

AI assistance

AI assistance was used to analyze the existing PRs, implement the compatibility backport, and draft tests. The submitting developer reviewed the changes and validation results.

Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@mergify mergify Bot added rust qwen Related to Qwen models labels Sep 11, 2026
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@biswapanda biswapanda changed the title [Rust] Support Dynamo Qwen3.5 multimodal generation on v0.28 [Rust] multimodal generation back-ported on vllm v0.28.0 Sep 11, 2026
@biswapanda
biswapanda marked this pull request as draft September 11, 2026 15:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

qwen Related to Qwen models rust

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant