Repository navigation
Load a Qwen3-VL checkpoint's language model as a diffusion prompt encoder - #987
Closed
vasilevklart wants to merge 4 commits into
Closed
vasilevklart wants to merge 4 commits into
vasilevklart wants to merge 4 commits into
Conversation
MiniMax-H3 conditions on hidden_states[50] of Qwen3-VL-32B's language model. For a text-only prompt that model is the scaffold's Qwen3 encoder: interleaved MRoPE with equal positions on its three axes rotates exactly as plain RoPE. Adds the config reader (Qwen3EncoderConfig.from_qwen3_vl), the model.language_model. key remap, a skip predicate that leaves the vision tower and the unused layers on disk, and an unfused q/k/v option: fused, cuBLAS rounds the 1024-wide k/v differently from transformers' three GEMMs. For prompts with images: Qwen3.5's vision tower moves to components/qwen_vision.py, with Qwen3-VL's deepstack mergers as an option, and the interleaved-MRoPE helpers to components/mrope.py. qwen3_5 imports both under its old names, binds its own resource key, keeps its rotary tables in the activations' dtype (transformers keeps fp32, the new default) and still refuses deepstack. A tower layer keeps an attention bound before when its node has no resource for it. vl_prompt.qwen3_vl_prompt splices the tower's tokens in at the image pads, positions the prompt in 3-D and hands the encoder its deepstack features.
No behaviour change. The encoder's docstring names Qwen3-VL, not the model stacked on it. The key-name test of the skip predicate goes: the parity tests load through it, and the loader refuses any key it lets through or drops. The tower tests share one setup, and the Qwen3.5 test checks the binding and the rounding, not the constructor's arguments.
Main's #1072 moved the Qwen3-VL ViT's shared parts into components/qwen3_vit.py and built Qwen3.5's and Qwen3-Omni's towers on it. qwen3_5/components/vision.py takes main's version; components/qwen_vision.py keeps only what Qwen3-VL adds on that ViT (its patch mergers and deepstack, fp32 rotary tables), the vision config Qwen3.5 shares, and FrameSDPA. The shared VisionAttention keeps its binding on a node that doesn't declare the resource, since the Qwen3-VL prompt encoder binds per-frame SDPA itself. Claude-Session: https://claude.ai/code/session_01EM3QYUVD2VuhDBCfazeGcy
Conflicts with #858 in the Qwen3 text encoder, both sides kept: main's cached rotary table, max_positions cap and host-known `padded`; this branch's fused_qkv choice and the Qwen3-VL inputs (spliced embeddings, caller-supplied 3-D MRoPE tables, deepstack). A caller's own rotary tables bypass the cached one; the cap applies either way. Claude-Session: https://claude.ai/code/session_01EM3QYUVD2VuhDBCfazeGcy
vasilevklart
changed the title
Collaborator
Author
|
Folded into #1155. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MiniMax-H3 (stacked on this) conditions on
hidden_states[50]of Qwen3-VL-32B's language model, with its keyframes in the prompt. The scaffold's Qwen3 prompt encoder only took Qwen3 text configs.The encoder now loads a Qwen3-VL checkpoint's language model:
Qwen3EncoderConfig.from_qwen3_vl, themodel.language_model.remap, and a skip predicate that leaves the tower and the unused layers on disk.vl_prompt.qwen3_vl_prompt: the tower's tokens at the image pads, 3-D MRoPE positions and deepstack features. A caller's MRoPE tables bypass the encoder's cached rotary table from Performance 1/7: shared engine, conductor and component changes #858.fused_qkv=Falseruns q/k/v as three GEMMs. Fused, cuBLAS rounds Qwen3-VL-32B's 1024-wide k/v differently from transformers.On top of the tower that #1072 moved to
components/qwen3_vit.py:components/qwen_vision.pyadds Qwen3-VL's deepstack mergers and fp32 rotary tables, as transformers keeps them.VisionAttention.bind_resourcestakes a caller's resource key.qwen3_5tocomponents/mrope.py.Qwen3.5 imports these under the old names and is otherwise unchanged: it still rounds its rotary tables to the activations' dtype and still refuses deepstack.
hidden_states[50]Tests:
test_qwen3_vl_text_encoder.py: a tiny random Qwen3-VL, whose tap matcheshidden_states[k]within 1e-5 for text, fused and unfused, and with two images of different grids.test_qwen_vision_tower.py: againstQwen3VLVisionModel, deepstack on and off, bit for bit in bf16; Qwen3.5 keeps its key, its rounding and a prior binding.6e5236766, main merged in at 3e98682, CPU Core fails only main's tests, by name. It merges cleanly with 9c05c56, where the H3, Qwen and scaffold CPU tests pass (270). ruff is clean.Not rerun: Qwen3.5's GPU image path (the captured block loop).
AI assistance (Claude) was used for this PR.