[CI] Split Distributed Tests (4 GPUs) into 3 parallel jobs

[khluu]

Summary

• Split the single ~1h09m "Distributed Tests (4 GPUs)" job into three smaller jobs targeting ~20m each
• Distributed Torchrun + Examples (4 GPUs) (~17m): torchrun tests + data_parallel + rlhf examples
• Distributed DP Tests (4 GPUs) (~20m): DP pytest tests (async_llm_dp, eagle_dp, external/internal/hybrid_lb, engine_core_client, test_utils)
• Distributed Compile + Comm (4 GPUs) (~25m): compile/fullgraph + pynccl + events + symm_mem + multiproc_executor

Test plan

• Verify all three new jobs pass in CI
• Confirm no tests are missing compared to the original single job
• Check that source_file_dependencies are correctly split across jobs

AI assistance was used (Claude). This is not duplicating any existing PR.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request is a great improvement for CI efficiency, splitting a large distributed test job into three smaller, parallel jobs. The overall structure of the split is logical. However, I've identified several issues with the source_file_dependencies in the new jobs. These include missing dependencies, incorrect file paths, and missing file extensions, which could prevent CI from running when relevant files are changed. Addressing these is critical for CI correctness.

[gemini-code-assist]

The source_file_dependencies for this job have a few issues that could cause it to not run when expected:

1. The job runs distributed/test_torchrun_example.py and distributed/test_torchrun_example_moe.py, but these are not listed as dependencies.
2. The dependency tests/examples/offline_inference/data_parallel.py seems incorrect. The command runs ../examples/offline_inference/data_parallel.py, so the dependency should likely be on examples/offline_inference/data_parallel.py.

Please update the dependencies to ensure this job is triggered correctly.

  - vllm/distributed/
  - tests/distributed/test_torchrun_example.py
  - tests/distributed/test_torchrun_example_moe.py
  - examples/offline_inference/rlhf.py
  - examples/offline_inference/rlhf_colocate.py
  - examples/offline_inference/new_weight_syncing/
  - examples/offline_inference/data_parallel.py

[gemini-code-assist]

The dependency tests/distributed/test_utils is missing the .py extension. The test file is tests/distributed/test_utils.py. Please correct the path to ensure changes to this file trigger the job.

  - tests/distributed/test_utils.py

[gemini-code-assist]

The dependencies tests/distributed/test_pynccl and tests/distributed/test_events are missing the .py extension. The test files are tests/distributed/test_pynccl.py and tests/distributed/test_events.py respectively. Please correct the paths to ensure changes to these files trigger the job.

  - tests/distributed/test_pynccl.py
  - tests/distributed/test_events.py

[khluu]

Distributed Torchrun + examples: 18m
Distributed DP Tests: 32m
Distributed Compile + Comm: 32m

Total is 82 minutes, 13 minutes more than original job due to overhead