[Feature][Core] Support Fabric detection to adapt the MNNVL protocol for the GB series by kebe7jun · Pull Request #33540 · vllm-project/vllm

kebe7jun · 2026-02-02T05:18:54Z

Purpose

Thank you very much for the solution provided by @tvegas1

Test Plan

Note that, currently, to use a custom allocator, sleep-mode must to be enabled.

# Prefill - Tray 0
export VLLM_NIXL_SIDE_CHANNEL_HOST=10.0.8.10 UCX_CUDA_IPC_ENABLE_MNNVL=y UCX_PROTO_INFO=y
vllm serve --gpu-memory-utilization=0.85 --tokenizer-mode=deepseek_v32 --no-enable-expert-parallel --model /home/ubuntu/DeepSeek-V3.2 --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_buffer_device":"cuda"}' --enable-sleep-mode -tp 2 -dp 4 -dpl 2 -dpa 10.0.8.10 -dpr 0

# Decode - Tray 2
export VLLM_NIXL_SIDE_CHANNEL_HOST=10.0.8.14 UCX_CUDA_IPC_ENABLE_MNNVL=y UCX_PROTO_INFO=y
vllm serve --gpu-memory-utilization=0.85 --tokenizer-mode=deepseek_v32 --no-enable-expert-parallel --model /home/ubuntu/DeepSeek-V3.2 --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_buffer_device":"cuda"}' --enable-sleep-mode -tp 2 -dp 4 -dpl 2 -dpa 10.0.8.14 -dpr 0

#Router
python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py --prefiller-hosts 10.0.8.10 --decoder-hosts 10.0.8.14 --prefiller-ports 8000 --decoder-ports 8000 --port 8000

Bench:

vllm bench serve --save-detailed --trust-remote-code --model /home/ubuntu/DeepSeek-V3.2 --seed 8961287 --dataset-name random --num-prompts 1024 --save-result --ignore-eos --max-concurrency 256 --random-input-len 4000 --random-output-len 1 --request-rate 32

I have tested it on both Hopper and Blackwell GPUs and it passed.

Test Result

In my env, the throughput increased threefold:

Main:

============ Serving Benchmark Result ============
Successful requests:                     1024
Failed requests:                         0
Maximum request concurrency:             256
Request rate configured (RPS):           32.00
Benchmark duration (s):                  249.39
Total input tokens:                      4096000
Total generated tokens:                  1024
Request throughput (req/s):              4.11
Output token throughput (tok/s):         4.11
Peak output token throughput (tok/s):    30.00
Peak concurrent requests:                285.00
Total token throughput (tok/s):          16428.22
---------------Time to First Token----------------
Mean TTFT (ms):                          50253.09
Median TTFT (ms):                        13394.40
P99 TTFT (ms):                           195786.07
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00
Median TPOT (ms):                        0.00
P99 TPOT (ms):                           0.00
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P99 ITL (ms):                            0.00
==================================================

This PR:

============ Serving Benchmark Result ============
Successful requests:                     1024
Failed requests:                         0
Maximum request concurrency:             256
Request rate configured (RPS):           32.00
Benchmark duration (s):                  58.77
Total input tokens:                      4096000
Total generated tokens:                  1024
Request throughput (req/s):              17.42
Output token throughput (tok/s):         17.42
Peak output token throughput (tok/s):    44.00
Peak concurrent requests:                287.00
Total token throughput (tok/s):          69707.43
---------------Time to First Token----------------
Mean TTFT (ms):                          11787.53
Median TTFT (ms):                        12994.56
P99 TTFT (ms):                           14845.10
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00
Median TPOT (ms):                        0.00
P99 TPOT (ms):                           0.00
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P99 ITL (ms):                            0.00
==================================================

The reason for the improved performance is that my GB200 environment is not configured with RoCE network, resulting in abnormal performance. But at this point, switching directly to the MNNVL protocol will become very stable

From the log, we can clearly see that it has been using cuda_ipc (MNNVL) as the cross-node transmission protocol.

[1770002852.416649] [XN-GB-Tray-10:3777218:0]   | ucp_context_0 inter-node cfg#2 | remote memory read by ucp_get*(multi) into cuda/GPU3 from cuda/dev[1]    |
[1770002852.416662] [XN-GB-Tray-10:3777218:0]   +--------------------------------+-------------------------------------------------------+------------------+
[1770002852.416662] [XN-GB-Tray-10:3777218:0]   |                              0 | copy-out                                              | rc_mlx5/mlx5_2:1 |
[1770002852.416662] [XN-GB-Tray-10:3777218:0]   |                         1..inf | zero-copy                                             | cuda_ipc/cuda    |
[1770002852.416662] [XN-GB-Tray-10:3777218:0]   +--------------------------------+-------------------------------------------------------+------------------+

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

gemini-code-assist

Code Review

This pull request adds support for detecting NVIDIA Fabric capabilities to adapt the MNNVL protocol, which should improve performance on systems with GB-series GPUs as shown by the benchmarks. The implementation correctly queries for fabric support and sets the appropriate memory allocation property. However, the new CUDA attributes used were introduced in CUDA 12.2. To maintain compatibility with older CUDA toolkits, the new code block should be conditionally compiled based on the CUDA version.

tvegas1 · 2026-02-02T09:26:35Z

@@ -115,6 +115,12 @@ void create_and_map(unsigned long long device, ssize_t size, CUdeviceptr d_mem,
  if (flag) {  // support GPUDirect RDMA if possible
    prop.allocFlags.gpuDirectRDMACapable = 1;
  }
+  int fab_flag = 0;
+  CUDA_CHECK(cuDeviceGetAttribute(
+      &fab_flag, CU_DEVICE_ATTRIBUTE_HANDLE_TYPE_FABRIC_SUPPORTED, device));


There are cases where although Fabric seems supported, the actual allocation fails, can be tested with program below. So we should probably either have fallback or maybe better environment variable.

// gcc -L$CUDA_HOME/lib64 -lcuda -I$CUDA_HOME/include fabri.c -lcuda #include <cuda.h> #include <stdio.h> #include <string.h> #define CHECK_CUDA(call) \ do { \ CUresult _status = (call); \ if (_status != CUDA_SUCCESS) { \ const char *err; \ cuGetErrorString(_status, &err); \ fprintf(stderr, "%s:%d: CUDA error: %s\n", __FUNCTION__, __LINE__, err); \ return -1; \ } \ } while (0) int main() { CHECK_CUDA(cuInit(0)); CUdevice dev; CHECK_CUDA(cuDeviceGet(&dev, 0)); int fabricSupported = 0; CHECK_CUDA(cuDeviceGetAttribute( &fabricSupported, CU_DEVICE_ATTRIBUTE_HANDLE_TYPE_FABRIC_SUPPORTED, dev)); if (!fabricSupported) { fprintf(stderr, "FABRIC handle type NOT supported on this device\n"); return 0; } CUcontext ctx; CHECK_CUDA(cuCtxCreate(&ctx, NULL, 0, dev)); const size_t size = 64 * 1024 * 1024; // 64 MB CUmemAllocationProp prop; memset(&prop, 0, sizeof(prop)); prop.type = CU_MEM_ALLOCATION_TYPE_PINNED; prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE; prop.location.id = dev; prop.requestedHandleTypes = CU_MEM_HANDLE_TYPE_FABRIC; CUmemGenericAllocationHandle handle; CHECK_CUDA(cuMemCreate(&handle, size, &prop, 0)); CUdeviceptr dptr; CHECK_CUDA(cuMemAddressReserve(&dptr, size, 0, 0, 0)); CHECK_CUDA(cuMemMap(dptr, size, 0, handle, 0)); CUmemAccessDesc accessDesc; memset(&accessDesc, 0, sizeof(accessDesc)); accessDesc.location.type = CU_MEM_LOCATION_TYPE_DEVICE; accessDesc.location.id = dev; accessDesc.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE; CHECK_CUDA(cuMemSetAccess(dptr, size, &accessDesc, 1)); printf("FABRIC pinned memory mapped at %p\n", (void*)dptr); CHECK_CUDA(cuMemUnmap(dptr, size)); CHECK_CUDA(cuMemAddressFree(dptr, size)); CHECK_CUDA(cuMemRelease(handle)); CHECK_CUDA(cuCtxDestroy(ctx)); return 0; }

I found this issue and discussed it with @youkaichao. I added a fallback strategy.

CU_DEVICE_ATTRIBUTE_HANDLE_TYPE_FABRIC_SUPPORTED is more like querying the cuda version to see if the cuda supports this functionality. it does not check if multi-node nvlink is available.

See https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__TYPES.html

CU_DEVICE_ATTRIBUTE_HANDLE_TYPE_FABRIC_SUPPORTED = 128
Device supports exporting memory to a fabric handle with cuMemExportToShareableHandle() or requested with cuMemCreate()

Looking at this attr's description, it should be for detecting the capabilities of the device, but indeed on the Hopper GPU, even when this value is 1, it cannot create CU_MEM_HANDLE_TYPE_FABRIC. Should we remove this detection? But it seems that doing this detection doesn't matter either.

youkaichao · 2026-02-02T10:42:00Z

 #endif

 #ifndef USE_ROCM
  // Allocate memory using cuMemCreate
-  CUDA_CHECK(cuMemCreate(p_memHandle, size, &prop, 0));
+  CUresult ret = (CUresult)cuMemCreate(p_memHandle, size, &prop, 0);
+  if (ret) {


can you fine-tune the error checking? only retry when ret is operation not permitted

these two errors can be related: CUDA_ERROR_NOT_PERMITTED and CUDA_ERROR_NOT_SUPPORTED

tvegas1 · 2026-02-02T10:45:21Z

I just started #33558 that contains C defines, vLLM environment and also activates KV cache with Fabric memory without sleep mode being needed. Feel free to take from it and put it here in this PR.

youkaichao · 2026-02-02T10:51:56Z

I just started #33558 that contains C defines, vLLM environment and also activates KV cache with Fabric memory without sleep mode being needed. Feel free to take from it and put it here in this PR.

@tvegas1 I'd prefer this PR without an environment variable (the auto-retry has better user experience). @kebe7jun will separate the code out to be independent of sleep mode.

tvegas1 · 2026-02-02T10:52:02Z

+    if (fab_flag && ret == cudaErrorNotPermitted) {
+      // Fabric allocation may fail on architectures other than Blackwell GPU,
+      // fallback to POSIX file descriptor
+      prop.requestedHandleTypes = CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR;


Just for my understanding, why falling back on CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR? Previously we would implicitly use CU_MEM_HANDLE_TYPE_NONE I think.

CU_MEM_HANDLE_TYPE_NONE disables any IPC, CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR makes it available for IPC via posix fd, which is available in all poxis systems.

tvegas1 · 2026-02-02T10:53:20Z

I just started #33558 that contains C defines, vLLM environment and also activates KV cache with Fabric memory without sleep mode being needed. Feel free to take from it and put it here in this PR.

@tvegas1 I'd prefer this PR without an environment variable (the auto-retry has better user experience). @kebe7jun will separate the code out to be independent of sleep mode.

I see, makes sense.

…for the GB series Signed-off-by: Kebe <mail@kebe7jun.com> Co-authored-by: Thomas Vegas <tvegas@nvidia.com>

Signed-off-by: youkaichao <youkaichao@gmail.com>

youkaichao

thanks for the fix!

…for the GB series (vllm-project#33540) Signed-off-by: Kebe <mail@kebe7jun.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Co-authored-by: Thomas Vegas <tvegas@nvidia.com> Co-authored-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Pai <416932041@qq.com>

…for the GB series (vllm-project#33540) Signed-off-by: Kebe <mail@kebe7jun.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Co-authored-by: Thomas Vegas <tvegas@nvidia.com> Co-authored-by: youkaichao <youkaichao@gmail.com> Signed-off-by: felix01.yu <felix01.yu@vipshop.com>

…for the GB series (vllm-project#33540) Signed-off-by: Kebe <mail@kebe7jun.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Co-authored-by: Thomas Vegas <tvegas@nvidia.com> Co-authored-by: youkaichao <youkaichao@gmail.com>

gemini-code-assist Bot reviewed Feb 2, 2026

View reviewed changes

Comment thread csrc/cumem_allocator.cpp

kebe7jun marked this pull request as draft February 2, 2026 07:21

tvegas1 reviewed Feb 2, 2026

View reviewed changes

tvegas1 mentioned this pull request Feb 2, 2026

[Core] Add option to allocate CUDA Fabric memory (required for MNNVL) #33558

Closed

5 tasks

kebe7jun force-pushed the fix/fabric-mem-type branch from 71b99a7 to d5222fd Compare February 2, 2026 10:30

kebe7jun marked this pull request as ready for review February 2, 2026 10:33

youkaichao reviewed Feb 2, 2026

View reviewed changes

kebe7jun force-pushed the fix/fabric-mem-type branch from d5222fd to 45d2d92 Compare February 2, 2026 10:50

tvegas1 reviewed Feb 2, 2026

View reviewed changes

[Feature][Core] Support Fabric detection to adapt the MNNVL protocol …

c972f13

…for the GB series Signed-off-by: Kebe <mail@kebe7jun.com> Co-authored-by: Thomas Vegas <tvegas@nvidia.com>

kebe7jun force-pushed the fix/fabric-mem-type branch from 45d2d92 to c972f13 Compare February 2, 2026 11:01

youkaichao reviewed Feb 2, 2026

View reviewed changes

Comment thread csrc/cumem_allocator.cpp Outdated

Update csrc/cumem_allocator.cpp

14b1167

Signed-off-by: youkaichao <youkaichao@gmail.com>

youkaichao approved these changes Feb 2, 2026

View reviewed changes

youkaichao enabled auto-merge (squash) February 2, 2026 12:01

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Feb 2, 2026

tvegas1 approved these changes Feb 2, 2026

View reviewed changes

tvegas1 mentioned this pull request Feb 2, 2026

[Question] How does nixl use MNNVL communication in the GB200 series environment? ai-dynamo/nixl#1240

Closed

youkaichao disabled auto-merge February 2, 2026 14:55

youkaichao merged commit 528e9b1 into vllm-project:main Feb 2, 2026
90 of 95 checks passed

kebe7jun deleted the fix/fabric-mem-type branch February 3, 2026 00:54

kebe7jun mentioned this pull request Feb 3, 2026

[Feature] Support manually enabling the cumem allocator #33648

Merged

5 tasks

haosdent mentioned this pull request Feb 27, 2026

[Bugfix] Fix Fabric/RDMA attribute queries poisoning global error_code in cumem allocator #35489

Closed

aimbit-ni mentioned this pull request Mar 5, 2026

[Bugfix] Guard fabric handle APIs for CUDA < 12.4 compatibility #36119

Closed

3 tasks

Uh oh!

Conversation

kebe7jun commented Feb 2, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

tvegas1 Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

kebe7jun Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

youkaichao Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

kebe7jun Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

youkaichao Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

youkaichao Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

tvegas1 commented Feb 2, 2026

Uh oh!

youkaichao commented Feb 2, 2026

Uh oh!

tvegas1 Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

youkaichao Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

kebe7jun Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tvegas1 commented Feb 2, 2026

Uh oh!

Uh oh!

youkaichao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kebe7jun commented Feb 2, 2026 •

edited by github-actions Bot

Loading

kebe7jun Feb 2, 2026 •

edited

Loading

kebe7jun Feb 2, 2026 •

edited

Loading

kebe7jun Feb 2, 2026 •

edited

Loading