Skip to content

Build native multimodal features before HTTP serialization - #193

Closed
xeophon wants to merge 1 commit into
mainfrom
feat/native-multimodal-features
Closed

xeophon wants to merge 1 commit into
mainfrom
feat/native-multimodal-features

Conversation

@xeophon

@xeophon xeophon commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Build vLLM multimodal features as native kwargs items and placeholder ranges. Server callers can pass them directly to mm_input; the token-generation HTTP client serializes them only when constructing its request. This removes the encode/decode round trip in prime-rl #3660 and consolidates the repeated hashes/placeholder assembly across model families.

Qwen3-VL, Qwen3.5, Gemma 4, and Nemotron 3.5 use the same field factories and tensor layouts. The HTTP request format stays the same. The public helper is renderers.client.build_mm_features.

Validation: 26 client tests passed; six vLLM-dependent tests were skipped on macOS. A CPU probe using the actual vLLM 0.31.0 field types and Qwen field factory verified native tensors, hashes, and placeholder ranges for all four renderer classes without importing the serialization module. Ruff passed; the repository's type check completed with its configured advisory diagnostics. No live engine was run.

Note

Build native multimodal kwargs before HTTP serialization in renderers.client

Model-specific multimodal builders (_build_qwen_vl_kwargs, _build_gemma4_kwargs, _build_nemotron35_kwargs) in client.py now return native MultiModalKwargsItems instead of serialized feature payloads. Hash and placeholder assembly plus kwargs-item encoding moved into generate, so serialization happens once at the HTTP-client boundary. An empty encoded kwargs mapping is sent as None.

  • Risk: behavior changes for empty-image requests — builders now return an empty item collection, and callers relying on the builders' serialized output must use generate's payload instead.

Macroscope summarized a3c71b3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant