Skip to content

feat(multimodal): graph Qwen2.5-VL custom encoder - #11565

Closed
furionw wants to merge 21 commits into
mainfrom
qiwa/qwen3-vl-custom-encoder
Closed

feat(multimodal): graph Qwen2.5-VL custom encoder#11565
furionw wants to merge 21 commits into
mainfrom
qiwa/qwen3-vl-custom-encoder

Conversation

@furionw

@furionw furionw commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Based on #11037.\n\n## Why\n\nDynamo’s custom encoder path can batch and cache image embeddings, but Qwen2.5-VL derives rotary and window metadata dynamically while segmented SDPA prevents efficient graph capture. Static grid metadata plus variable-length FlashAttention makes its complete 32-block vision tower capturable while preserving native vision parity. A separate performance experiment pairs the 3B vision tower with Qwen2.5-1.5B by truncating its completed 2048-wide output to 1536; this is untrained and makes no quality or model-parity claim. The Qwen3 encoder remains available for comparison.\n\n## Effect\n\nPerformance-only closed-loop H100 image sweep, 1,000 requests per cell:\n\n| Concurrency | TTFT avg (ms) | E2E avg (ms) | Throughput (req/s) |\n| ---: | ---: | ---: | ---: |\n| 8 | 102.8 | 501.2 | 15.865 |\n| 16 | 549.8 | 897.3 | 17.688 |\n| 32 | 1109.9 | 1519.6 | 20.734 |\n\n## What Change\n\n- Graph Qwen2.5 rotary, window attention, merger, and output reordering across static buckets.\n- Preserve Qwen3, caches, coalescing, pinned transfers, and bounded micro-batching.\n- Add audited H100 QPS and closed-loop concurrency benchmark workflows.\n\n## Test Plan\n\n- 36 focused H100 tests and all targeted pre-commit hooks passed.\n- Graph parity/allocator verification and audited 3,000-request concurrency sweep passed.

@github-actions github-actions Bot added feat documentation Improvements or additions to documentation backend::vllm Relates to the vllm backend labels Jul 11, 2026
@github-actions

github-actions Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

@datadog-official

This comment has been minimized.

@furionw furionw changed the title feat(multimodal): add Qwen3-VL custom encoder repro feat(multimodal): graph Qwen3-VL custom encoder batches Jul 11, 2026
@github-actions github-actions Bot added the frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` label Jul 12, 2026
@furionw
furionw force-pushed the qiwa/qwen3-vl-custom-encoder branch from eae1fde to 0d8d773 Compare July 12, 2026 05:22
@furionw
furionw force-pushed the qiwa/vision-encoder-batcher branch from 492569d to 7286f09 Compare July 14, 2026 02:39
Add a supervised, thread-affine micro-batcher and route AsyncVisionEncoder forwards through it. Preserve preprocess atomicity while making lifecycle, failures, and teardown deterministic.

Signed-off-by: Qi Wang <qiwa@nvidia.com>
@furionw
furionw force-pushed the qiwa/vision-encoder-batcher branch from 7286f09 to 54e4cc5 Compare July 14, 2026 02:46
@copy-pr-bot

copy-pr-bot Bot commented Jul 14, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@furionw
furionw force-pushed the qiwa/qwen3-vl-custom-encoder branch from 2df6ef4 to 931a1da Compare July 14, 2026 04:54
@furionw furionw changed the title feat(multimodal): graph Qwen3-VL custom encoder batches feat(multimodal): graph Qwen2.5-VL custom encoder batches Jul 14, 2026
furionw added 5 commits July 13, 2026 23:58
Signed-off-by: furionw <qiwa@nvidia.com>
Signed-off-by: furionw <qiwa@nvidia.com>
Signed-off-by: furionw <qiwa@nvidia.com>
Signed-off-by: furionw <qiwa@nvidia.com>
@furionw furionw changed the title feat(multimodal): graph Qwen2.5-VL custom encoder batches feat(multimodal): graph Qwen2.5-VL custom encoder Jul 14, 2026
Use vLLM variable-length FlashAttention for the Qwen2.5-VL custom encoder and launch its decoder in language-model-only mode. Restore the historical Qwen3-VL encoder module so both implementations remain available.

Signed-off-by: furionw <qiwa@nvidia.com>
@furionw
furionw force-pushed the qiwa/qwen3-vl-custom-encoder branch from 28907d9 to 78e74f6 Compare July 14, 2026 17:51
furionw added 2 commits July 14, 2026 15:15
Signed-off-by: furionw <qiwa@nvidia.com>
Signed-off-by: furionw <qiwa@nvidia.com>
@furionw
furionw force-pushed the qiwa/vision-encoder-batcher branch from 54e4cc5 to 0392783 Compare July 15, 2026 00:53
Base automatically changed from qiwa/vision-encoder-batcher to main July 15, 2026 18:48
@furionw furionw closed this Jul 17, 2026
@furionw
furionw deleted the qiwa/qwen3-vl-custom-encoder branch July 17, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend documentation Improvements or additions to documentation feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant