[feat]pd disaggregated support cross-machine

[weiguihua2]

What this PR does / why we need it?

pd disaggregated support cross-machine.
We send the primary and secondary node information of node p to node d. When node d pulls the KV data, it retrieves the corresponding primary or secondary node information from the mapping.

Does this PR introduce any user-facing change?

How was this patch tested?

• vLLM version: v0.12.0
• vLLM main: vllm-project/vllm@ad32e3e

[github-actions]

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

• A PR should do only one thing, smaller PRs enable faster reviews.
• Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
• Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

[gemini-code-assist]

Code Review

This pull request introduces multi-node KV cache transfer capabilities by implementing a handshake mechanism. Key changes include adding KVConnectorHandshakeMetadata and local_ip to MooncakeAgentMetadata, and remote_multi_nodes_meta_mapping to ReqMeta to facilitate cross-node metadata exchange. New methods get_handshake_metadata and set_xfer_handshake_metadata were added to MooncakeConnector and MooncakeConnectorScheduler respectively, along with a multi_nodes_meta_mapping attribute in the scheduler. The MooncakeConnectorWorker now generates and stores its own handshake metadata, including its local IP, and uses a new helper method _get_remote_host_info_by_port to resolve remote host information based on the exchanged metadata. The WorkerV1 class was updated to expose this KV connector handshake metadata. A review comment identified an issue where the default value for remote_multi_nodes_meta_mapping was incorrectly set to an integer 1 instead of an empty dictionary {}, which would lead to an AttributeError.

[gemini-code-assist]

The default value for remote_multi_nodes_meta_mapping is set to 1, which is incorrect for a parameter that is expected to be a dictionary. If the remote_multi_nodes_meta_mapping key is not present in kv_transfer_params, this will cause an AttributeError: 'int' object has no attribute 'get' in _get_remote_host_info_by_port when it tries to access the mapping. The default value should be an empty dictionary {}.

Suggested change

	remote_multi_nodes_meta_mapping=kv_transfer_params.get("remote_multi_nodes_meta_mapping", 1),
	remote_multi_nodes_meta_mapping=kv_transfer_params.get("remote_multi_nodes_meta_mapping", {}),

[lidenghui1110]

Could you please add more explain of why you need this remote_multi_nodes_meta_mapping?

As I know, when prefill node crossing multi-node, each MooncakeConnectorScheduler will add it's own remote_host and remote_engine_id to kv_transfer_params, no matter master or slave node, which is set here.

[weiguihua2]

Could you please add more explain of why you need this remote_multi_nodes_meta_mapping?

As I know, when prefill node crossing multi-node, each MooncakeConnectorScheduler will add it's own remote_host and remote_engine_id to kv_transfer_params, no matter master or slave node, which is set here.

For DP cross-machine scenarios, there will be multiple instances, but for TP cross-machine scenarios (such as Ray cross-machine or MP cross-machine), there will only be one instance. In this case, the information of the master and slave nodes needs to be sent to the D node.

[lidenghui1110]

Could you please add more explain of why you need this remote_multi_nodes_meta_mapping?
As I know, when prefill node crossing multi-node, each MooncakeConnectorScheduler will add it's own remote_host and remote_engine_id to kv_transfer_params, no matter master or slave node, which is set here.

For DP cross-machine scenarios, there will be multiple instances, but for TP cross-machine scenarios (such as Ray cross-machine or MP cross-machine), there will only be one instance. In this case, the information of the master and slave nodes needs to be sent to the D node.

I got it. But the problem is only existing on Ray cross-machine scenario with only one dp rank like pure TP? While using MP cross-machine, each node will have a DPEnginecore with a MooncakeConnectorScheduler, it will follow kv_transfer_params, each node can set its own remote_host.

Could please confirm if I am right or wrong?

[weiguihua2]

Could you please add more explain of why you need this remote_multi_nodes_meta_mapping?
As I know, when prefill node crossing multi-node, each MooncakeConnectorScheduler will add it's own remote_host and remote_engine_id to kv_transfer_params, no matter master or slave node, which is set here.

For DP cross-machine scenarios, there will be multiple instances, but for TP cross-machine scenarios (such as Ray cross-machine or MP cross-machine), there will only be one instance. In this case, the information of the master and slave nodes needs to be sent to the D node.

I got it. But the problem is only existing on Ray cross-machine scenario with only one dp rank like pure TP? While using MP cross-machine, each node will have a DPEnginecore with a MooncakeConnectorScheduler, it will follow kv_transfer_params, each node can set its own remote_host.

Could please confirm if I am right or wrong?
The community merged a new feature a few weeks ago, which is a TP cross-machine not based on Ray; there is only one DPEnginecore with a MooncakeConnectorScheduler across multiple nodes.
vllm-project/vllm#23691

[weiguihua2]

Could you please add more explain of why you need this remote_multi_nodes_meta_mapping?
As I know, when prefill node crossing multi-node, each MooncakeConnectorScheduler will add it's own remote_host and remote_engine_id to kv_transfer_params, no matter master or slave node, which is set here.

For DP cross-machine scenarios, there will be multiple instances, but for TP cross-machine scenarios (such as Ray cross-machine or MP cross-machine), there will only be one instance. In this case, the information of the master and slave nodes needs to be sent to the D node.

I got it. But the problem is only existing on Ray cross-machine scenario with only one dp rank like pure TP? While using MP cross-machine, each node will have a DPEnginecore with a MooncakeConnectorScheduler, it will follow kv_transfer_params, each node can set its own remote_host.
Could please confirm if I am right or wrong?
The community merged a new feature a few weeks ago, which is a TP cross-machine not based on Ray; there is only one DPEnginecore with a MooncakeConnectorScheduler across multiple nodes.
vllm-project/vllm#23691

For the DP cross-machine, each node will have a DPEnginecore with a MooncakeConnectorScheduler, it will follow kv_transfer_params, each node can set its own remote_host.The current code is compatible with this scenario.

[lidenghui1110]

Could you please add more explain of why you need this remote_multi_nodes_meta_mapping?
As I know, when prefill node crossing multi-node, each MooncakeConnectorScheduler will add it's own remote_host and remote_engine_id to kv_transfer_params, no matter master or slave node, which is set here.

For DP cross-machine scenarios, there will be multiple instances, but for TP cross-machine scenarios (such as Ray cross-machine or MP cross-machine), there will only be one instance. In this case, the information of the master and slave nodes needs to be sent to the D node.

I got it. But the problem is only existing on Ray cross-machine scenario with only one dp rank like pure TP? While using MP cross-machine, each node will have a DPEnginecore with a MooncakeConnectorScheduler, it will follow kv_transfer_params, each node can set its own remote_host.
Could please confirm if I am right or wrong?
The community merged a new feature a few weeks ago, which is a TP cross-machine not based on Ray; there is only one DPEnginecore with a MooncakeConnectorScheduler across multiple nodes.
vllm-project/vllm#23691

For the DP cross-machine, each node will have a DPEnginecore with a MooncakeConnectorScheduler, it will follow kv_transfer_params, each node can set its own remote_host.The current code is compatible with this scenario.

Got it. Thanks for your explanation.

[LCAIZJ]

For the DP cross-machine, primary and secondary node in prefill can be distinguished by engine id, so is this PR specifically addressing the MP scenario?

[weiguihua2]

For the DP cross-machine, primary and secondary node in prefill can be distinguished by engine id, so is this PR specifically addressing the MP scenario?

Yes, DP cross-machine has multiple engine cores, while TP cross-machine has only one engine core.