[Core] Avoid unnecessary coordination for non-MoE data parallel

[ZJY0516]

Purpose

FIX #24461
Avoid unnecessary coordination for non-MoE data parallel

Test Plan

vllm serve /data/datasets/models-hf/Qwen3-4B-Instruct-2507-FP8/ --served-model-name qwen -dp 2 --max-model-len 10240

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[ZJY0516]

CC @njhill

[gemini-code-assist]

Code Review

This pull request introduces an optimization to avoid unnecessary dummy batch execution for non-MoE models in a data-parallel setup, which should reduce overhead. The changes are well-structured, adding a skip_dummy_batch path that is conditionally used based on whether expert parallelism is enabled. The implementation correctly preserves the necessary synchronization for data parallelism while skipping the actual model execution. I've added one comment regarding device placement to prevent potential future bugs. The changes look good overall.

[ZJY0516]

@njhill Should I also consider the offline inference scenario?

[njhill]

Thanks for this @ZJY0516! I don't think it looks quite like what we had in mind though. We want to avoid doing the additional collectives altogether and ideally avoid initializing the associated torch distributed process groups.

We should also be able to avoid the messaging done for coordination of request waves, etc.

[ZJY0516]

@njhill But if self.get_dp_padding is not executed, non-moe model on dp > 1 will be blocked forever. Do you have any suggstion?

[njhill]

@njhill But if self.get_dp_padding is not executed, non-moe model on dp > 1 will be blocked forever. Do you have any suggstion?

I'm not sure that I follow, the different ranks should be completely independent in the non-MoE case...

[ZJY0516]

@njhill But if self.get_dp_padding is not executed, non-moe model on dp > 1 will be blocked forever. Do you have any suggstion?

I'm not sure that I follow, the different ranks should be completely independent in the non-MoE case...

I think it's because there is a all_reduce operation in DPMetadata.num_tokens_across_dp which is called in self.get_dp_padding.

[ZJY0516]

PR #24105 changed the all_reduce operation from CPU to GPU, which introduced additional cudaMemcpy operations.

Now(free gpu utilization will increase to 100%)

Before that PR(free gpu utilization 0%)

@njhill How should we address this? Should we avoid the memory copy to reduce GPU utilization?

[njhill]

@ZJY0516 sorry, I was thinking of something different for this. Basically most aspects should work like a non-DP deployment - we don't need the DP process group, etc. The DPCoordinator doesn't need to synchronize request waves, etc.

[ZJY0516]

@ZJY0516 sorry, I was thinking of something different for this. Basically most aspects should work like a non-DP deployment - we don't need the DP process group, etc. The DPCoordinator doesn't need to synchronize request waves, etc.

You mean for non-moe model we don't need the DP process group and synchronize request waves?