[1/N] Elastic EP Milestone 2

[itayalroy]

This PR completes the work in #26278 originally authored by @libertyeagle, who designed and implemented the core architecture for elastic EP milestone 2. In collaboration with @tlrmchlsmth, we rebased the PR on top of latest vLLM main, resolved all conflicts and remaining issues to help get it ready for merge. All elastic EP features (scale up, scale down, and serving requests between scaling events) have been tested with multiple EP backends.

See Elastic EP RFC for more details: #20323

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[gemini-code-assist]

Code Review

This pull request introduces a significant new feature, Elastic Expert Parallelism, with extensive changes across the codebase. The core logic for state management and distributed communication has been heavily modified. While the overall architecture seems well-thought-out, I've identified a few critical issues related to distributed coordination and configuration that could lead to deadlocks or incorrect behavior during scaling operations. Specifically, there are potential deadlocks in the scale-down logic, incorrect calculations for resource allocation, and assumptions about parallelism dimensions that may not hold true. These issues need to be addressed to ensure the stability and correctness of the new elastic EP feature.

[gemini-code-assist]

There is a potential deadlock condition during the scale-down process. After the collective call to _eplb_reshuffle_before_scale_down, the execution paths for remaining and removing engines diverge. The _progress_removing_engine calls _destroy_old_comm_groups(), while _progress_remaining_engine calls _create_standby_groups(). Both of these methods trigger collective RPCs.

Since the workers are in different states and initiating different collective operations, this will lead to a deadlock, as not all participants of the original group will be part of the same collective call. To fix this, all workers in the old distributed groups must participate in the same collective operations in the same order until the groups are destroyed or the removing workers are definitively excluded from future collectives.

[gemini-code-assist]

The reshape operation for all_ranks in create_standby_groups does not account for prefill_context_model_parallel_size (pcp_size). It seems to assume pcp_size is always 1. If pcp_size > 1 is used with elastic EP, this will result in an incorrect tensor shape and subsequent errors when creating the standby process groups. The reshape operation should include prefill_context_model_parallel_size to correctly handle all parallelism dimensions.

Suggested change

	all_ranks = torch.arange(new_world_size_across_dp).reshape(
	-1, new_dp_size, pp_size, tp_size
	)
	all_ranks = torch.arange(new_world_size_across_dp).reshape(
	-1, new_dp_size, pp_size, self.prefill_context_model_parallel_size, tp_size
	)

[mergify]

Hi @itayalroy, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @itayalroy, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @itayalroy, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @itayalroy.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork