[NIXL][OOT platform] support nixl_connector with oot platform and other nixl_backend by xuechendi · Pull Request #25121 · vllm-project/vllm

xuechendi · 2025-09-18T01:24:52Z

Purpose

Originally, nixl_connector only supports in-tree platforms and UCX as backend. In this PR, we want to extend ths nixl_connector support to OOT platform as well. Also, want to introduce the option to use different NIXL_backend instead of UCX for vllm

Test Plan

use existing test

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

gemini-code-assist

Code Review

This pull request extends the nixl_connector to support Out-Of-Tree (OOT) platforms and allows configuring the NIXL backend via an environment variable. The changes are well-structured, following the existing platform interface pattern in vLLM by adding new methods to the Platform class. This makes the integration clean and extensible. I have one suggestion to improve the robustness of handling the new environment variable.

vllm/envs.py

xuechendi · 2025-09-18T01:27:38Z

@robertgshaw2-redhat @NickLucche please help to review this PR, thanks

chaunceyjiang · 2025-09-18T03:42:04Z

vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

                                                self.nixl_memory_type)
        logger.debug("Registering descs: %s", caches_data)
-        self.nixl_wrapper.register_memory(descs)
+        self.nixl_wrapper.register_memory(descs, backends=[self.nixl_backend])


I think using --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_connector_extra_config":{"backend":"ucx"}}' might be more appropriate.

NickLucche

Hey @xuechendi , thanks a lot for contributing!

This looks clean, but I also agree with @chaunceyjiang it would be even cleaner to put extra configs in kv_connector_extra_config":{"backend":"ucx"}} instead of adding more env vars.

Also, it would be quite useful to the community if you could add a line or two to the docs highlighting your change, as well as a simple unit test to test_nixl_connector.py checking memory_type/kv_buffer_type are as intended (we have more changes lined up to it in review).
Thank you!

docs/serving/expert_parallel_deployment.md

xuechendi · 2025-09-19T02:23:56Z

@NickLucche, for adding a new UT in test_nixl_connector.py, I added one "test_kv_buffer_to_nixl_memory_types", which is initialized with a FakePlatform, may you check if this is what you expected?

chaunceyjiang · 2025-09-19T06:21:40Z

vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

@@ -63,6 +64,8 @@
    "tpu": ("cpu", ),
    "xpu": ("cpu", ),
 }
+# support for oot platform by providing mapping in current_platform
+_NIXL_SUPPORTED_DEVICE.update(current_platform.get_nixl_supported_devices())


The variable _NIXL_SUPPORTED_DEVICE appears to be unused.

It is used here:
https://github.com/vllm-project/vllm/blob/main/vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py#L476-L482

mergify · 2025-09-19T06:22:13Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @xuechendi.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

xuechendi · 2025-09-22T14:12:02Z

@NickLucche @chaunceyjiang , may you help to check again, or please approve if the PR is looking good to you, Thanks

NickLucche

LGTM, thanks @xuechendi !

Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

…er nixl_backend (vllm-project#25121) Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

…er nixl_backend (vllm-project#25121) Signed-off-by: Chendi Xue <Chendi.Xue@intel.com> Signed-off-by: charlifu <charlifu@amd.com>

…er nixl_backend (#25121) Signed-off-by: Chendi Xue <Chendi.Xue@intel.com> Signed-off-by: yewentao256 <zhyanwentao@126.com>

…er nixl_backend (vllm-project#25121) Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

mergify · 2025-11-26T11:00:05Z

Documentation preview: https://vllm--25121.org.readthedocs.build/en/25121/

xuechendi requested review from ApostaC and NickLucche as code owners September 18, 2025 01:24

gemini-code-assist bot reviewed Sep 18, 2025

View reviewed changes

vllm/envs.py Outdated Show resolved Hide resolved

chaunceyjiang reviewed Sep 18, 2025

View reviewed changes

NickLucche requested changes Sep 18, 2025

View reviewed changes

mergify bot added the kv-connector label Sep 18, 2025

xuechendi requested a review from hmellor as a code owner September 19, 2025 01:26

mergify bot added the documentation Improvements or additions to documentation label Sep 19, 2025

chaunceyjiang reviewed Sep 19, 2025

View reviewed changes

docs/serving/expert_parallel_deployment.md Outdated Show resolved Hide resolved

mergify bot added the v1 label Sep 19, 2025

xuechendi requested a review from NickLucche September 19, 2025 03:13

mergify bot added the ci/build label Sep 19, 2025

xuechendi force-pushed the dev/nixl_plugin branch from cec89f3 to c76b975 Compare September 19, 2025 03:34

chaunceyjiang reviewed Sep 19, 2025

View reviewed changes

mergify bot added the needs-rebase label Sep 19, 2025

xuechendi force-pushed the dev/nixl_plugin branch from c76b975 to 2b365f3 Compare September 19, 2025 19:54

mergify bot removed the needs-rebase label Sep 19, 2025

NickLucche approved these changes Sep 22, 2025

View reviewed changes

NickLucche enabled auto-merge (squash) September 22, 2025 16:49

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 22, 2025

auto-merge was automatically disabled September 22, 2025 19:38
Head branch was pushed to by a user without write access

xuechendi force-pushed the dev/nixl_plugin branch from 6578346 to 200ae4c Compare September 22, 2025 19:38

xuechendi added 4 commits September 22, 2025 22:56

update nixl_connector with backend option

fab251a

Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

use kv_connector_extra_config instead of env for backend setting

22734d0

Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

update doc with backend option

95540a0

Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

update backend to backends

a8377a7

Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

xuechendi added 3 commits September 22, 2025 22:56

Add UT to test kv_buffer_device to nixl_memory_type mapping

fe50002

Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

Fix mypy

7e467cc

Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

retrigger CI

77147bc

Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

xuechendi force-pushed the dev/nixl_plugin branch 2 times, most recently from ae425a2 to 6e21b38 Compare September 22, 2025 23:01

jikunshang enabled auto-merge (squash) September 23, 2025 00:00

fix CI for missing of nixl_agent_config

5a78f48

Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

auto-merge was automatically disabled September 23, 2025 00:43
Head branch was pushed to by a user without write access

xuechendi force-pushed the dev/nixl_plugin branch from 6e21b38 to 5a78f48 Compare September 23, 2025 00:43

jikunshang enabled auto-merge (squash) September 23, 2025 00:49

Fix CI

5b0745c

Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

auto-merge was automatically disabled September 23, 2025 02:03
Head branch was pushed to by a user without write access

jikunshang enabled auto-merge (squash) September 23, 2025 03:17

jikunshang merged commit 5774b0a into vllm-project:main Sep 23, 2025
46 checks passed

xuechendi mentioned this pull request Sep 24, 2025

[Cuda2CPU][P/D] Add cuda2cpu support in NixlConnector #24690

Merged

FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request Sep 25, 2025

[NIXL][OOT platform] support nixl_connector with oot platform and oth…

8ad964c

…er nixl_backend (vllm-project#25121) Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

yewentao256 pushed a commit that referenced this pull request Oct 3, 2025

[NIXL][OOT platform] support nixl_connector with oot platform and oth…

921945c

…er nixl_backend (#25121) Signed-off-by: Chendi Xue <Chendi.Xue@intel.com> Signed-off-by: yewentao256 <zhyanwentao@126.com>

choprahetarth pushed a commit to Tandemn-Labs/vllm that referenced this pull request Oct 11, 2025

[NIXL][OOT platform] support nixl_connector with oot platform and oth…

8fa18fb

…er nixl_backend (vllm-project#25121) Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

lywa1998 pushed a commit to lywa1998/vllm that referenced this pull request Oct 20, 2025

[NIXL][OOT platform] support nixl_connector with oot platform and oth…

2e8ad90

…er nixl_backend (vllm-project#25121) Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

rtourgeman pushed a commit to rtourgeman/vllm that referenced this pull request Nov 10, 2025

[NIXL][OOT platform] support nixl_connector with oot platform and oth…

6a8d95e

…er nixl_backend (vllm-project#25121) Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>

Uh oh!

Conversation

xuechendi commented Sep 18, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

xuechendi commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chaunceyjiang Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

xuechendi Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

NickLucche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xuechendi commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chaunceyjiang Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

xuechendi Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Sep 19, 2025

Uh oh!

xuechendi commented Sep 22, 2025

Uh oh!

NickLucche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mergify bot commented Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xuechendi commented Sep 18, 2025 •

edited by github-actions bot

Loading

xuechendi commented Sep 18, 2025 •

edited

Loading

xuechendi commented Sep 19, 2025 •

edited

Loading