[Model] enable data parallel for Llama4 vision encoder

[jennyyyyzhen]

Summary:
Llama4 vision encoder in dp8 is ~3x as fast as in tp8, especially when handling a large number of input images (eg. 9 images per request).
Add an enable_vision_encoder_data_parallel to allow using different parallelism for vision model and language model.

Unit test
pytest tests/models/multimodal/generation/test_common.py -k "llama4"

MM Eval
Baseline TP8

lm_eval --model vllm-vlm --model_args pretrained=meta-llama/Llama-4-Scout-17B-16E-Instruct,tensor_parallel_size=8,max_model_len=32768,gpu_memory_utilization=0.9 --tasks chartqa --batch_size auto --apply_chat_template

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
chartqa	0	none	0	anywhere_accuracy	↑	0.8872	±	0.0063
		none	0	exact_match	↑	0.6548	±	0.0095
		none	0	relaxed_accuracy	↑	0.8848	±	0.0064

DP8

lm_eval --model vllm-vlm --model_args pretrained=meta-llama/Llama-4-Scout-17B-16E-Ins
truct,tensor_parallel_size=8,max_model_len=32768,gpu_memory_utilization=0.9,enable_multimodal_encoder_data_parallel=true --tasks chartqa --batch_size auto --apply_chat_template

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
chartqa	0	none	0	anywhere_accuracy	↑	0.8852	±	0.0064
		none	0	exact_match	↑	0.6492	±	0.0095
		none	0	relaxed_accuracy	↑	0.8820	±	0.0065

perf result

HF_CHECKPOINT=meta-llama/Llama-4-Scout-17B-16E-Instruct
 python benchmarks/benchmark_serving.py --backend openai-chat --model $HF_CHECKPOINT --dataset-name hf --dataset-path lmarena-ai/VisionArena-Chat --hf-split train --num-prompts 1000 --endpoint /v1/chat/completions --max-concurrency 32 --ignore-eos --seed 0

DP8

============ Serving Benchmark Result ============
Successful requests:                     1000      
Benchmark duration (s):                  115.99    
Total input tokens:                      87321     
Total generated tokens:                  128000    
Request throughput (req/s):              8.62      
Output token throughput (tok/s):         1103.55   
Total Token throughput (tok/s):          1856.38   
---------------Time to First Token----------------
Mean TTFT (ms):                          641.44    
Median TTFT (ms):                        571.65    
P99 TTFT (ms):                           2443.98   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          23.86     
Median TPOT (ms):                        23.52     
P99 TPOT (ms):                           31.16     
---------------Inter-token Latency----------------
Mean ITL (ms):                           23.69     
Median ITL (ms):                         17.91     
P99 ITL (ms):                            222.95    
==================================================

Baseline TP8

============ Serving Benchmark Result ============
Successful requests:                     1000      
Benchmark duration (s):                  124.06    
Total input tokens:                      87321     
Total generated tokens:                  128000    
Request throughput (req/s):              8.06      
Output token throughput (tok/s):         1031.74   
Total Token throughput (tok/s):          1735.58   
---------------Time to First Token----------------
Mean TTFT (ms):                          758.51    
Median TTFT (ms):                        734.38    
P99 TTFT (ms):                           1805.42   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.95     
Median TPOT (ms):                        24.71     
P99 TPOT (ms):                           30.54     
---------------Inter-token Latency----------------
Mean ITL (ms):                           24.76     
Median ITL (ms):                         17.87     
P99 ITL (ms):                            283.55    
==================================================

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[sarckk]

this looks good to me overall, could you also add MM eval results with DP and TP?

[DarkLight1337]

cc @houseroad

[ywang96]

Sorry for the delayed review and thank you for the contribution! Overall and I left some comments!

[ywang96]

This is great! As a follow-up I think it makes sense to rewrite this into a separate function to be shared by other models since this is not specific to mllama4 vision encoder in particular!

[houseroad]

maybe create an issue to follow up?

[houseroad]

Looks pretty reasonable. Could you rebase and address @ywang96 's feedback?

[houseroad]

wondering if we need clone here?

[houseroad]

Looks good to me.

[houseroad]

we can add some unittest for function. (good for a follow up PR)

[jennyyyyzhen]

@houseroad follow up PR is here #19103

[houseroad]

nit: we can add some sanity checks to ensure _consolidate_qkv_weights operate appropriately.

[houseroad]

@ywang96 , could you give another pass?

[ywang96]

Please address Lu's comment otherwise LGTM!