feat(rollout): vLLM prefill/decode (PD) disaggregation via static vllm-router

[aoshen02]

What

Carves the PD (prefill/decode) disaggregation half out of #108 so that PR can stay DP+EP-only. On top of the colocate DP+EP rollout, this adds the prefill/decode split via the production vllm-router in static mode:

• vllm_engine.py — for a rollout group whose worker_type is prefill/decode (node_rank 0), pin the NIXL side-channel VLLM_NIXL_SIDE_CHANNEL_HOST/PORT to the orchestrator-allocated disaggregation_bootstrap_port, plumbed through _compute_server_args and persisted on the actor. The KV connector (--kv-transfer-config) itself is supplied by the operator via normal vLLM arg passthrough.
• rollout.py — _start_router gains a static-PD path (bind + static prefill_urls/decode_urls, health check disabled). start_rollout_servers reserves the router endpoint, starts engines (which do not self-register), collects per-worker_type URLs, then launches the router in PD mode.

NIXL pull mode → no per-URL bootstrap port

vime uses vLLM's NIXL pull connector: the prefill engine returns the side-channel coords (remote_host/remote_port/remote_block_ids) to the decode side at request time via the response's kv_transfer_params. The router never consumes a per-prefill-URL bootstrap port (that path is Mooncake-only). So prefill_urls are (url, None) tuples — matching SkyRL's static-PD reference and vllm_router._parse_prefill_urls (which explicitly accepts none).

Relationship to #108

• feat(vllm): DP+EP and PD (prefill/decode) rollout disaggregation via the production vllm-router #108 is DP+EP-only; this PR is the PD layer on top. The two diffs touch disjoint regions of build_vllm_cmd_and_env, so they compose without conflict.

Test

• py_compile clean on both files.
• Reviewed with Codex: no correctness issues in the bootstrap-port plumbing or the (url, None) tuple shape. One pre-existing latent crash surfaced in the refactored _start_router (the router-reuse early-return returned a 2-tuple while callers unpack 3) and is fixed here (returns (ip, port, None)).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces support for Prefill/Decode (PD) disaggregation in the vLLM engine by launching prefill and decode engines with the NIXL KV connector and configuring unique side-channel ports. It also adds a static PD router startup flow in the rollout module that collects engine URLs before launching the router. The review feedback suggests replacing a process-alive assertion with an explicit conditional check to avoid issues when assertions are disabled in production, and optimizing the sequential ray.get calls inside a loop by batching them into a single parallelized call.

[gemini-code-assist]

Using assert for production control flow or process health checks is a bad practice because assertions can be disabled globally in Python when run with the -O (optimize) flag. If assertions are disabled, assert process.is_alive() will be completely ignored, and if the process failed to start or crashed immediately, the code will proceed silently. Use an explicit conditional check and raise a RuntimeError instead.

Suggested change

	assert process.is_alive()
	if not process.is_alive():
	raise RuntimeError("Failed to start vLLM-router process.")

[gemini-code-assist]

Gathering Ray ObjectRefs inside a nested loop and calling ray.get sequentially on each iteration is a performance bottleneck. It forces sequential round-trips to the Ray actors, blocking the main thread. Instead, you should collect all remote calls into a list and resolve them in a single parallelized ray.get call.

        if use_static_pd_router:
            prefill_engines = []
            decode_engines = []
            for g in server_groups:
                for e in g.engines:
                    if e is None:
                        continue
                    if g.worker_type == "prefill":
                        prefill_engines.append(e)
                    elif g.worker_type == "decode":
                        decode_engines.append(e)

            prefill_refs = []
            for e in prefill_engines:
                prefill_refs.extend([e.get_url.remote(), e.get_pd_bootstrap_port.remote()])
            decode_refs = [e.get_url.remote() for e in decode_engines]

            all_results = ray.get(prefill_refs + decode_refs)

            prefill_results = all_results[:len(prefill_refs)]
            decode_results = all_results[len(prefill_refs):]

            prefill_urls: list[tuple] = []
            for i in range(0, len(prefill_results), 2):
                url = prefill_results[i]
                bport = prefill_results[i+1]
                if url:
                    prefill_urls.append((url, bport))

            decode_urls: list[str] = [url for url in decode_results if url]
            _launch_static_pd_router(args, router_ip, router_port, prom_port, prefill_urls, decode_urls)

[CalvinXKY]

We’d better test it e2e.

[aoshen02]

Cross-Node PD Validation Results (H200 ×2)

Setup

• Training: PP=2, TP=4, CP=2, EP=8, alltoall dispatcher — 2× H200 (16 GPUs total)
• Rollout PD: prefill 8 GPUs (h200-0) + decode 8 GPUs (h200-1), TP=2 per engine (4+4 engines), NIXL KV transfer
• Model: Qwen3-30B-A3B (BF16), dapo-math-17k training data
• Eval: AIME-2024 (30 problems, n_samples=1, top_k=1, max_response_len=16384)
• Colocated (train + rollout share GPUs)
• Image: inferactinc/public:vime-latest (vLLM 0.22.0)

Runs

Job	Config	Iterations	Status
413	TP=2, mem_util=0.5	2	✅ Completed (manually stopped)
418	TP=2, mem_util=0.7	3	✅ Completed (manually stopped)
419	TP=4, mem_util=0.7	2	✅ Completed (manually stopped)

Eval Before Train (iteration 0)

• AIME-2024 accuracy: 63.3% (19/30)
• Mean response length: 12,272 tokens
• Truncated ratio: 36.7% (11/30 hit max_response_len=16384)

Steady-State Performance (iteration 1, no eval overhead)

Metric	PD TP=2	PD TP=4
rollout_time	55.6s	52.9s
train_wait	98.3s	101.3s
update_weights	32.9s	36.9s
wake_up	0.8s	1.0s
log_probs	13.5s	13.2s
actor_train	26.2s	25.8s
tokens/GPU/s	~470	~476

TP=2 is the sweet spot — faster weight sync offsets the slightly slower per-engine throughput.

Issues Encountered (all environment-level, no vime code bugs)

1. Gloo 127.0.1.1 loopback — Ubuntu /etc/hosts + --network host → fix: GLOO_SOCKET_IFNAME=ens7
2. Ray Redis stale session — previous job's host-network containers left Redis running → fix: pre-flight container cleanup in harness
3. disable_health_check removed — already fixed by this PR (feat(rollout): vLLM prefill/decode (PD) disaggregation via static vllm-router #166) with disable_circuit_breaker

Conclusion

Cross-node PD disaggregation via NIXL works correctly on H200 ×2. No new code bugs found. The disable_circuit_breaker fix in this PR is validated end-to-end.

[CalvinXKY]

LGTM