[Feature]vLLM multi-node rollout engine topology

[CalvinXKY]

Summary

• Add vLLM multi-node rollout topology: one Ray actor per node-rank, head owns HTTP/router, workers launch with --headless.
• Consolidate launch configuration into vllm_engine.py (compute_server_args → build_vllm_cmd_and_env → launch_server_process).
• Multi-node CLI: --nnodes, --node-rank, --master-*, --data-parallel-backend mp, --distributed-executor-backend mp.
• Rename rollout per-group overrides to vllm_overrides; remove SGLang references from vllm_utils.

Part of #65 (orchestration layer; multinode integration tests and rollout cleanup in follow-up PRs).

Test plan

Completed

• Local: pre-commit on changed files (ruff, black, isort, autoflake) — all passed
• A800-server / vime_v22 container: pytest -q tests/unit/backends/vllm_utils/ → 60 passed
  • Topology: single-node (rollout_num_gpus_per_engine=4, num_gpus_per_node=8, TP=4) and multi-node rank split (rollout_num_gpus_per_engine=16, num_gpus_per_node=8 → nnodes=2, TP=8, PP=2)
  • CLI flags: --nnodes, --node-rank, --master-addr, --master-port, --headless (worker), --data-parallel-backend mp, --distributed-executor-backend mp
  • Single-node path: no extra distributed flags appended when nnodes=1

Single-node backward compatibility (existing workloads — not re-run in this PR)

These configs should behave identically to pre-PR behavior because nnodes=1 when rollout_num_gpus_per_engine <= num_gpus_per_node; no multi-node CLI flags are injected.

• tests/test_vllm_generate_endpoint.py — Qwen3-0.6B (1 GPU): compute_server_args + launch_server_process → /inference/v1/generate
• tests/test_vllm_generate_endpoint.py — Qwen3-30B-A3B (1 GPU): MoE rollout + logprob capture
• tests/test_vllm_generate_endpoint.py — Qwen3-30B-A3B + R3 routing replay (1 GPU): rollout_routed_experts shape check
• Existing single-host training scripts (e.g. Qwen3-30B-A3B, 8 GPU colocate, rollout_num_gpus_per_engine=8, nnodes_per_engine=1) — unchanged code path; head-only HTTP/router/weight-update logic

Multi-node orchestration (supported by this PR — integration validation deferred to 2/N)

This PR adds the Ray-actor topology and vLLM launch flags; cross-host E2E is planned in follow-up PRs (see PR #66 sweep matrix). Target configs:

Config	Model	Hosts	rollout_num_gpus_per_engine	num_gpus_per_node	nnodes_per_engine	vLLM TP × PP	What this exercises
s3	Qwen3-30B-A3B	2 × 8 GPU	16	8	2	16 × 1	One logical engine spans both hosts; worker --headless; mp backends
s4	Qwen3-30B-A3B	2 × 8 GPU	8	4	2	8 × 1	Multi-engine + nnodes=2 per engine
s9	Qwen3-30B-A3B	2 × 8 GPU	16	8	2	8 × 2	Cross-host vLLM PP=2 × nnodes=2 (eager; CUDA-graph capture hang is a known vLLM PP issue)

Planned test artifacts (2/N, aligned with #65 / PR #66):

• tests/test_qwen3_30B_A3B_pr66_sweep.py — parametrized 11-config × 2-step sweep
• tests/multinode/{head,worker,launch_cross_host}.sh — cross-host Ray + container orchestration

Part of #65

[gemini-code-assist]

Code Review

This pull request refactors the vLLM rollout backend to support multi-node configurations by introducing a structured VllmEngineTopology and separating server argument computation from process launching. It also cleans up various HTTP endpoints and adds corresponding unit and integration tests. The feedback highlights three key issues: first, _wait_worker_process_alive incorrectly blocks the main thread for the full timeout duration when the worker process is healthy; second, the override checks for vllm_data_parallel_backend and vllm_distributed_executor_backend are dead code because these fields are in SKIPPED_DESTS; and third, consecutive checks for the colocate attribute in build_vllm_subprocess_env should be consolidated for better readability.

[gemini-code-assist]

The current implementation of _wait_worker_process_alive will block the main thread for the entire duration of timeout_s (defaulting to 300.0 seconds / 5 minutes) if the worker process is healthy and remains alive. This introduces a mandatory 5-minute delay on every multi-node launch for non-head nodes.

Since the head node's _wait_server_healthy already handles the full model loading and cluster synchronization health check, the worker nodes only need a brief startup check (e.g., 10 seconds) to ensure the subprocess didn't immediately crash due to invalid arguments or environment issues.

We should reduce the default timeout_s to a much smaller value like 10.0 seconds.

Suggested change

	def _wait_worker_process_alive(process: multiprocessing.Process, timeout_s: float = 300.0) -> None:
	def _wait_worker_process_alive(process: multiprocessing.Process, timeout_s: float = 10.0) -> None:

[gemini-code-assist]

In arguments.py, data_parallel_backend and distributed_executor_backend are added to SKIPPED_DESTS. This means they are skipped during parser construction and do not exist in get_vllm_cli_action_table().

As a result, _user_overrode(args, "vllm_data_parallel_backend") and _user_overrode(args, "vllm_distributed_executor_backend") will always return False because get_vllm_cli_action_table().get(dest) returns None.

If these backends are strictly orchestrator-controlled and should never be overridden by the user, the _user_overrode checks here are dead code and can be removed. If they are meant to be overridable, they should not be in SKIPPED_DESTS or the override detection logic needs to be adjusted.

[gemini-code-assist]

The getattr(args, "colocate", False) check is performed twice consecutively. We can consolidate these checks into a single conditional block to improve readability and maintainability.

Suggested change

	if getattr(args, "colocate", False):
	import slime
	vime_root = os.path.dirname(os.path.dirname(os.path.abspath(slime.__file__)))
	existing_pp = env.get("PYTHONPATH", "")
	if vime_root not in {p for p in existing_pp.split(os.pathsep) if p}:
	env["PYTHONPATH"] = os.pathsep.join(filter(None, [vime_root, existing_pp]))
	if getattr(args, "colocate", False):
	env.setdefault("VLLM_ALLOW_INSECURE_SERIALIZATION", "1")
	if getattr(args, "colocate", False):
	import slime
	vime_root = os.path.dirname(os.path.dirname(os.path.abspath(slime.__file__)))
	existing_pp = env.get("PYTHONPATH", "")
	if vime_root not in {p for p in existing_pp.split(os.pathsep) if p}:
	env["PYTHONPATH"] = os.pathsep.join(filter(None, [vime_root, existing_pp]))
	env.setdefault("VLLM_ALLOW_INSECURE_SERIALIZATION", "1")

[CalvinXKY]

Multi-node validation on A800-server + A800-server2 (vime_v22)

Environment

• Hosts: A800-server (7.216.199.149) + A800-server2 (7.216.196.62)
• Container: vime_v22 on both nodes (shared NFS mount /data/nfs_87)
• vLLM: 0.21.1rc1.dev38+gff712f644
• Model: Qwen3-30B-A3B copied to container-local disk (/root/local_models/Qwen3-30B-A3B, 57G, 16 shards) to avoid slow NFS weight load

Connectivity pre-check

• Management SSH 7.216.199.149 ↔ 7.216.196.62: PASS

s3 — cross-host colocate, nnodes_per_engine=2 ✅ PASS

Field	Value
Config	s3
Model	Qwen3-30B-A3B
Hosts	2 × 8 GPU (16 GPU logical engine)
rollout_num_gpus_per_engine	16
num_gpus_per_node	8
nnodes_per_engine	2
vLLM TP × PP	16 × 1
dist_init_addr	7.216.199.149:38271

Head log (excerpt)

[node_rank=0] config=s3 nnodes=2 local_gpus=8 tp=16 pp=1 headless=False dist_init_addr=7.216.199.149:38271
non-default args: {..., 'distributed_executor_backend': 'mp', 'master_addr': '7.216.199.149', 'master_port': 38271, 'nnodes': 2, 'tensor_parallel_size': 16, ...}
DP group leader: node_rank=0, ..., world_size=16, local_world_size=8
[head] server healthy, running generate
PASS config=s3 response_len=16 status=Status.TRUNCATED tokens_tail=[220, 18, 488, 220]

Worker log (excerpt)

[node_rank=1] config=s3 nnodes=2 local_gpus=8 tp=16 pp=1 headless=True dist_init_addr=7.216.199.149:38271
Launching vLLM headless multiproc executor, with head node address 7.216.199.149:38271
world_size=16 rank=8..15 local_rank=0..7 distributed_init_method=tcp://7.216.199.149:38271
PASS worker config=s3 stayed alive until test-done

[CalvinXKY]

A800 dual-node multinode smoke test results

Environment

• Head: A800-server (7.216.199.149) · Worker: A800-server2 (7.216.196.62)
• Container: vime_v22 on both nodes
• Model: Qwen3-30B-A3B (local copy at /root/local_models/Qwen3-30B-A3B)
• vLLM: 0.21.1rc1.dev38+gff712f644

Config	TP×PP	rollout_num_gpus_per_engine	num_gpus_per_node	Result	Log
s3	16×1	16	8	✅ PASS	logs/pr68_s3_local4/head.log
s4	8×1	8	4	✅ PASS	logs/pr68_s4_20260529_131533/head.log
s9	8×2 (enforce_eager)	16	8	✅ PASS	logs/pr68_s9_20260529_142251/head.log

s9 excerpt (cross-host PP=2, world_size=16, generate returned 200 OK):

[node_rank=0] config=s9 nnodes=2 local_gpus=8 tp=8 pp=2 headless=False dist_init_addr=7.216.199.149:54313
(APIServer pid=311) INFO: ... "POST /inference/v1/generate HTTP/1.1" 200 OK
PASS config=s9 response_len=16 status=Status.TRUNCATED tokens_tail=[220, 18, 488, 220]

Notes

• s9 requires enforce_eager=True (same MoE+PP CUDA graph issue as PR Mirror SGLang rollout architecture for vLLM #66).
• Barrier sync fix: head no longer deletes worker-ready after worker touches it.
• Manual test scripts live under NFS multinode_test/ (not committed to the vime repo).
• Occasional TCPStore Broken pipe warnings on the worker after teardown are benign; PASS is determined by PASS config=... in head/worker logs.

[CalvinXKY]

A800 dual-node end-to-end training validation (follow-up)

How this differs from the earlier smoke tests

The inference smoke tests above only exercise the rollout / inference side:

• Standalone vLLM engine launch across 2 nodes (s3 / s4 / s9 topologies)
• One-shot POST /inference/v1/generate through the router
• Validates cross-node TP/PP wiring, Ray barrier sync, and worker registration

This E2E run exercises the full RL training loop on the same hardware and configs:

• train.py submitted as a Ray job on 2 nodes × 8 GPUs
• Megatron actors load the torch_dist checkpoint and run 2 rollout steps (num-rollout=2)
• Each step: rollout → reward → IPC weight sync (train → inference) → Megatron PPO update
• Confirms the orchestration path used in real training, not just isolated inference

Same model (Qwen3-30B-A3B), same container image, same s3/s4/s9 engine layouts as the smoke tests.

Results

Config	TP×PP	Rollout layout	E2E steps	Result
s3	16×1	1 engine, TP16 across both nodes	2	✅ PASS
s4	8×1	2 engines, TP8 each (4 GPU/node)	2	✅ PASS
s9	8×2 (enforce_eager)	2 engines, TP8 PP2 (16 GPU/engine)	2	✅ PASS

Pass criteria: Ray job completed successfully; both training steps finished; update_weights between rollout and train had no fatal errors; grad_norm showed no NaN/inf (small TIGHT batch — grad_norm=0 is expected).

Observed behavior (s9): first step is slow (~2–3 min) while Megatron loads the distributed checkpoint and performs the initial weight sync into vLLM; this is normal, not a hang.

Code change uncovered during E2E

While running s9, E2E hit missing/invalid PP CLI wiring. Fixed in arguments.py (dffb2da):

• Add --vllm-pp-size and validate via getattr
• Exclude vllm_pp_size from generic orchestration forwarding (was incorrectly emitted as --pp-size); build_vllm_cmd_and_env sends --pipeline-parallel-size instead

[CalvinXKY]

Single-node R3 training validation (follow-up)

How this differs from the multinode E2E above

The multinode E2E tests validate cross-host rollout topologies (s3/s4/s9) with 2 nodes × 8 GPUs.

This follow-up checks that the original single-machine training recipes still work on the same A800 hardware — in particular routing replay (R3) with vLLM:

• One node, 8 GPUs, Ray head only (no cross-host vLLM TP/PP)
• Full train.py path: rollout → R3 routed experts → IPC weight sync → Megatron update
• Two layouts from our run_script r3 work:
  • disagg 4+4 — 4 train + 4 rollout GPUs (same as qwen3-30B-A3B-4t4i-r3-300step.sh)
  • colocate — train + rollout share all 8 GPUs (--colocate)

Each layout only needs to pass on one machine (no need to duplicate on both nodes).

Results

Mode	GPUs	Steps	Result	Notes
disagg (4+4)	4 train + 4 rollout, TP1 EP4	2	✅ PASS	Matches run_script r3 layout; --use-rollout-routing-replay
colocate	8 shared, TP4 EP8	1	✅ PASS	1-step smoke; 2-step colocate hits host RAM limit on 1TB nodes during weight sync

Pass criteria: Ray job succeeded; training step(s) completed with R3 enabled; no fatal update_weights errors.

Together with the inference smoke + multinode E2E above, this covers single-node r3, cross-node rollout, and cross-node full training loop on Qwen3-30B-A3B.

[CalvinXKY]

Cross-node Expert Parallel validation (follow-up)

Follow-up on the dual-node A800 setup (2×8 GPUs, vime_v22, Qwen3-30B-A3B, branch feature/multi_nodes). Two independent EP test groups — no vime code changes required; validation scripts only.

How this differs from prior tests

Prior test	What it covered
Inference smoke (s3/s4/s9)	Cross-host vLLM TP/PP topologies, generate-only
Multinode E2E (s3/s4/s9)	Full train loop with Megatron TP=8 / EP=8 / CP=2, but rollout side was dense TP (no vLLM EP)
Single-node R3	Routing replay on one machine

This follow-up explicitly validates MoE expert parallelism spanning both nodes on each stack:

Test A — vLLM cross-node EP (inference)

Config	Nodes	vLLM layout	EP	Result
vllm_ep16	2×8	TP=8, DP=2	16	✅ PASS

Head runs one generate after cross-node vLLM startup (--vllm-enable-expert-parallel, --vllm-data-parallel-size 2). Worker stays alive headless until teardown.

Test B — Megatron cross-node EP (E2E, 2 steps)

Config	Actor	Megatron	Rollout vLLM	Result
megatron_ep8_xnode	2×8	TP=8, EP=8, CP=2	TP=16 cross-node (no vLLM EP)	✅ PASS

Full colocate path: rollout → IPC weight sync → train × 2 steps. grad_norm=0 on both steps; no NaN/inf.

Note on EP=16: A megatron_ep16 (TP=1, EP=16) attempt hung at load_checkpoint — the existing torch_dist checkpoint was converted on 8 GPUs (TP4/EP8) and reshard to EP=16 does not complete in practice. EP=8 across both nodes is sufficient to validate Megatron cross-node EP with the current checkpoint; full EP=16 would need a 16-GPU checkpoint convert.

Summary

Stack	Cross-node EP	Code changes	Result
vLLM	EP=16 (TP8×DP2)	None	✅ PASS
Megatron	EP=8 (TP8/CP2)	None	✅ PASS

Together with inference smoke, multinode E2E, and single-node R3, this covers vLLM cross-node EP inference and Megatron cross-node EP training on Qwen3-30B-A3B without additional changes on feature/multi_nodes.