Test test_low_latency.py failed on H100 with ROCE

[ImbaPlayer]

Issue Description

I'm working on an H100 GPU cluster with RoCE drivers properly installed on the network interface cards.
While the test_intranode.py script runs successfully and produces expected results, the test_low_latency consistently fails with errors.
Technical Details:

• NVSHMEM version installed: 3.1.7-1 (following the README instructions)
• Suspected compatibility issue: Potential mismatch between NVSHMEM version and RoCE configuration

I would greatly appreciate any assistance or insights to resolve this. Below are the specific error messages for reference:

Actual Result

(base) root@ubuntu:/work/DeepEP-main# python tests/test_low_latency.py
local_rank:1, ip:127.0.0.1 port:3004
world_size:2, rank:1
local_rank:0, ip:127.0.0.1 port:3004
world_size:2, rank:0
setting....
setting....
setted...
rank:0, num_ranks:2
Allocating buffer size: 2116.292096 MB ...

/work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 ibv_modify_qp failed
ibv_modify_qp failed


/work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1432: non-zero status: 7 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1432: non-zero status: 7 ep_connect failed
ep_connect failed


/work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1499: non-zero status: 7 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1499: non-zero status: 7 transport create connect failed
transport create connect failed


/work/nvshmem_src_3.1.7-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed
connect EPS failed


/work/nvshmem_src_3.1.7-1/nvshmem_src/src/host/init/init.cu:1001: non-zero status: 7 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/host/init/init.cu:1001: non-zero status: 7 nvshmem setup connections failed
nvshmem setup connections failed


/work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 ibv_modify_qp failed

/work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1432: non-zero status: 7 ep_connect failed

/work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1499: non-zero status: 7 transport create connect failed

/work/nvshmem_src_3.1.7-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed
/work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110
ibv_modify_qp failed
/work/nvshmem_src_3.1.7-1/nvshmem_src/src/host/init/init.cu:1001: non-zero status: 7
nvshmem setup connections failed
/work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1432: non-zero status: 7
ep_connect failed
/work/nvshmem_src_3.1.7-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1074:
nvshmem initialization failed, exiting
/work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1499: non-zero status: 7
transport create connect failed

/work/nvshmem_src_3.1.7-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed

/work/nvshmem_src_3.1.7-1/nvshmem_src/src/host/init/init.cu:1001: non-zero status: 7 nvshmem setup connections failed

/work/nvshmem_src_3.1.7-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1074: nvshmem initialization failed, exiting

W0303 20:33:45.904000 12712 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 12777 via signal SIGTERM
Traceback (most recent call last):
  File "/work/DeepEP-main/tests/test_low_latency.py", line 164, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 204, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 1 terminated with exit code 255

Env

Ubuntu2204

Linux ubuntu 5.15.0-25-generic #25-Ubuntu SMP Wed Mar 30 15:54:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

nvidia-smi

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.05              Driver Version: 560.35.05      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          Off |   00000000:18:00.0 Off |                    0 |
| N/A   22C    P0             68W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          Off |   00000000:2A:00.0 Off |                    0 |
| N/A   25C    P0             71W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA H100 80GB HBM3          Off |   00000000:3A:00.0 Off |                    0 |
| N/A   24C    P0             69W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA H100 80GB HBM3          Off |   00000000:5D:00.0 Off |                    0 |
| N/A   22C    P0             70W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA H100 80GB HBM3          Off |   00000000:9A:00.0 Off |                    0 |
| N/A   24C    P0             71W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA H100 80GB HBM3          Off |   00000000:AB:00.0 Off |                    0 |
| N/A   25C    P0             72W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA H100 80GB HBM3          Off |   00000000:BA:00.0 Off |                    0 |
| N/A   23C    P0             70W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA H100 80GB HBM3          Off |   00000000:DB:00.0 Off |                    0 |
| N/A   22C    P0             69W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

ibv_devinfo

hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      a088:c203:0059:910c
        sys_image_guid:                 a088:c203:0059:910c
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_1
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      a088:c203:0050:a72c
        sys_image_guid:                 a088:c203:0050:a72c
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_2
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      a088:c203:007e:1dba
        sys_image_guid:                 a088:c203:007e:1dba
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_3
        transport:                      InfiniBand (0)
        fw_ver:                         16.35.4030
        node_guid:                      e8eb:d303:0055:750a
        sys_image_guid:                 e8eb:d303:0055:750a
        vendor_id:                      0x02c9
        vendor_part_id:                 4119
        hw_ver:                         0x0
        board_id:                       MT_0000000425
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_4
        transport:                      InfiniBand (0)
        fw_ver:                         16.35.4030
        node_guid:                      e8eb:d303:0055:750b
        sys_image_guid:                 e8eb:d303:0055:750a
        vendor_id:                      0x02c9
        vendor_part_id:                 4119
        hw_ver:                         0x0
        board_id:                       MT_0000000425
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_5
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      a088:c203:0060:27e6
        sys_image_guid:                 a088:c203:0060:27e6
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_6
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      a088:c203:007e:1c3a
        sys_image_guid:                 a088:c203:007e:1c3a
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_7
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      a088:c203:0060:2b1e
        sys_image_guid:                 a088:c203:0060:2b1e
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_8
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      a088:c203:007d:ab62
        sys_image_guid:                 a088:c203:007d:ab62
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_9
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      a088:c203:007d:ab9a
        sys_image_guid:                 a088:c203:007d:ab9a
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

[sphish]

This is a bug in NVSHMEM 3.1.7 and can be resolved by using NVSHMEM 3.2.5.
#17 (comment)

[Baibaifan]

This is a bug in NVSHMEM 3.1.7 and can be resolved by using NVSHMEM 3.2.5. #17 (comment)

How to deal with conflicts in deepep patch packages?

[sphish]

This is a bug in NVSHMEM 3.1.7 and can be resolved by using NVSHMEM 3.2.5. #17 (comment)

How to deal with conflicts in deepep patch packages?

For the current patch, the conflict is caused by a commit modifying the CMake file. You can skip this commit. I will later upload a new patch compatible with NVSHMEM 3.2.5.

[Baibaifan]

This is a bug in NVSHMEM 3.1.7 and can be resolved by using NVSHMEM 3.2.5. #17 (comment)

How to deal with conflicts in deepep patch packages?

For the current patch, the conflict is caused by a commit modifying the CMake file. You can skip this commit. I will later upload a new patch compatible with NVSHMEM 3.2.5.

I tried to modify the CMakeLists.txt file, but CMakeLists.txt has been refactored, and I don't know how to modify the original patch 140 and 165. I hope you can give me some suggestions. I'll wait for you to update the patch. @sphish

[sphish]

I have created a new branch that updates NVSHMEM to version 3.2.5. However, I don't have a RoCE environment for verification. Can you test this out? @Baibaifan
Branch: https://github.com/deepseek-ai/DeepEP/tree/roce-support

[Baibaifan]

I have created a new branch that updates NVSHMEM to version 3.2.5. However, I don't have a RoCE environment for verification. Can you test this out? @Baibaifan Branch: https://github.com/deepseek-ai/DeepEP/tree/roce-support

ok, I will try it and does the cmake configuration need to be modified? @sphish

CUDA_HOME=/path/to/cuda \
GDRCOPY_HOME=/path/to/gdrcopy \
NVSHMEM_SHMEM_SUPPORT=0 \
NVSHMEM_UCX_SUPPORT=0 \
NVSHMEM_USE_NCCL=0 \
NVSHMEM_MPI_SUPPORT=0 \
NVSHMEM_IBGDA_SUPPORT=1 \
NVSHMEM_PMIX_SUPPORT=0 \
NVSHMEM_TIMEOUT_DEVICE_POLLING=0 \
NVSHMEM_USE_GDRCOPY=1 \
cmake -S . -B build/ -DCMAKE_INSTALL_PREFIX=/path/to/your/dir/to/install

[sphish]

I have created a new branch that updates NVSHMEM to version 3.2.5. However, I don't have a RoCE environment for verification. Can you test this out? @Baibaifan Branch: https://github.com/deepseek-ai/DeepEP/tree/roce-support

ok, I will try it and does the cmake configuration need to be modified? @sphish

CUDA_HOME=/path/to/cuda \
GDRCOPY_HOME=/path/to/gdrcopy \
NVSHMEM_SHMEM_SUPPORT=0 \
NVSHMEM_UCX_SUPPORT=0 \
NVSHMEM_USE_NCCL=0 \
NVSHMEM_MPI_SUPPORT=0 \
NVSHMEM_IBGDA_SUPPORT=1 \
NVSHMEM_PMIX_SUPPORT=0 \
NVSHMEM_TIMEOUT_DEVICE_POLLING=0 \
NVSHMEM_USE_GDRCOPY=1 \
cmake -S . -B build/ -DCMAKE_INSTALL_PREFIX=/path/to/your/dir/to/install

Nope.

[kunfupanda-hw]

I have created a new branch that updates NVSHMEM to version 3.2.5. However, I don't have a RoCE environment for verification. Can you test this out? @Baibaifan Branch: https://github.com/deepseek-ai/DeepEP/tree/roce-support

I have the same problem on one 8*H100 cluster node for test_low_latency.py, and update NVSHMEM to version 3.2.5 with the newest patch. However, it doesn't work, it lacks for, with the same logs, i.e., /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 ibv_modify_qp failed ibv_modify_qp failed. However, I have tried the ib_write_bw cmd successfully. Any kind suggestion for this problem?

[sphish]

I have created a new branch that updates NVSHMEM to version 3.2.5. However, I don't have a RoCE environment for verification. Can you test this out? @Baibaifan Branch: https://github.com/deepseek-ai/DeepEP/tree/roce-support

I have the same problem on one 8*H100 cluster node for test_low_latency.py, and update NVSHMEM to version 3.2.5 with the newest patch. However, it doesn't work, it lacks for, with the same logs, i.e., /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 ibv_modify_qp failed ibv_modify_qp failed. However, I have tried the ib_write_bw cmd successfully. Any kind suggestion for this problem?

Could you provide the logs from your NVSHMEM 3.2.5 run? It appears you're still using NVSHMEM 3.1.7.

[Baibaifan]

I have created a new branch that updates NVSHMEM to version 3.2.5. However, I don't have a RoCE environment for verification. Can you test this out? @Baibaifan Branch: https://github.com/deepseek-ai/DeepEP/tree/roce-support

ok, I will try it and does the cmake configuration need to be modified? @sphish

CUDA_HOME=/path/to/cuda \
GDRCOPY_HOME=/path/to/gdrcopy \
NVSHMEM_SHMEM_SUPPORT=0 \
NVSHMEM_UCX_SUPPORT=0 \
NVSHMEM_USE_NCCL=0 \
NVSHMEM_MPI_SUPPORT=0 \
NVSHMEM_IBGDA_SUPPORT=1 \
NVSHMEM_PMIX_SUPPORT=0 \
NVSHMEM_TIMEOUT_DEVICE_POLLING=0 \
NVSHMEM_USE_GDRCOPY=1 \
cmake -S . -B build/ -DCMAKE_INSTALL_PREFIX=/path/to/your/dir/to/install

Nope.

Hi @sphish, The process works, but the performance does not seem to meet expectations.

env:

1. H100 80GB HBM3 *8/HPC
2. 4 * CX7 NICs 400gb/s/HPC
3. RoCE

rank0
NCCL_DEBUG=INFO MASTER_ADDR=xxx WORLD_SIZE=2 RANK=0 python tests/test_internode.py

rank1
NCCL_DEBUG=INFO MASTER_ADDR=xxx WORLD_SIZE=2 RANK=1 python tests/test_internode.py

[tuning] Best dispatch (FP8): SMs 24, NVL chunk 28, RDMA chunk 8: 4.09 GB/s (RDMA), 13.36 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 28, RDMA chunk 32: 4.57 GB/s (RDMA), 14.91 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 3, RDMA chunk 32: 2.97 GB/s (RDMA), 9.69 GB/s (NVL)

The difference between the above speed and the official data is greater than the difference between the network cards. How can I test the performance with perftest? Is there any reference?

[kunfupanda-hw]

I have created a new branch that updates NVSHMEM to version 3.2.5. However, I don't have a RoCE environment for verification. Can you test this out? @Baibaifan Branch: https://github.com/deepseek-ai/DeepEP/tree/roce-support

I have the same problem on one 8*H100 cluster node for test_low_latency.py, and update NVSHMEM to version 3.2.5 with the newest patch. However, it doesn't work, it lacks for, with the same logs, i.e., /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 ibv_modify_qp failed ibv_modify_qp failed. However, I have tried the ib_write_bw cmd successfully. Any kind suggestion for this problem?

Could you provide the logs from your NVSHMEM 3.2.5 run? It appears you're still using NVSHMEM 3.1.7.

sorry for that old logs. The newest logs are as below, with some my comment logs.

(base) root@ubuntu:/work/DeepEP-main# python tests/test_low_latency.py 
local_rank:5, ip:127.0.0.1 port:3004
world_size:8, rank:5
local_rank:4, ip:127.0.0.1 port:3004
world_size:8, rank:4
local_rank:6, ip:127.0.0.1 port:3004
world_size:8, rank:6
local_rank:7, ip:127.0.0.1 port:3004
world_size:8, rank:7
local_rank:0, ip:127.0.0.1 port:3004
world_size:8, rank:0
local_rank:1, ip:127.0.0.1 port:3004
world_size:8, rank:1
local_rank:2, ip:127.0.0.1 port:3004
world_size:8, rank:2
local_rank:3, ip:127.0.0.1 port:3004
world_size:8, rank:3
setting....
setting....
setting....
setting....
setting....
setting....
setting....
setting....
setted...
rank:2, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
setted...
rank:0, num_ranks:8
Allocating buffer size: 2116.2912 MB ...
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
setted...
rank:1, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
setted...
rank:5, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
setted...
rank:4, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
after cpp.Buffer
setted...
rank:7, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
setted...
rank:3, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
after cpp.Buffer
setted...
rank:6, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 
ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 ep_connect failed 
transport create connect failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 
connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem setup connections failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 
ibv_modify_qp failed 
ibv_modify_qp failed 

ibv_modify_qp failed 
ibv_modify_qp failed 



/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed 
ep_connect failed 
ep_connect failed 

ep_connect failed 
ep_connect failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 

transport create connect failed 

transport create connect failed 
transport create connect failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 
connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 


connect EPS failed 

connect EPS failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 nvshmem setup connections failed 
nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 nvshmem setup connections failed 


nvshmem setup connections failed 

nvshmem setup connections failed 



/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 
ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed 
ep_connect failed 
ep_connect failed 

ep_connect failed 



/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 ep_connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 

transport create connect failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 transport create connect failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 ibv_modify_qp failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 ibv_modify_qp failed 
ibv_modify_qp failed 
connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 transport create connect failed 

connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 


connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
ep_connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 ep_connect failed 

ep_connect failed 
nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem setup connections failed 
connect EPS failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 

nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: transport create connect failed 

transport create connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 transport create connect failed 

nvshmem initialization failed, exiting 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: 
nvshmem initialization failed, exiting 
nvshmem setup connections failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 nvshmem initialization failed, exiting 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 nvshmem initialization failed, exiting 
connect EPS failed 


connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: connect EPS failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem initialization failed, exiting 

nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem setup connections failed 
nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: 

nvshmem initialization failed, exiting 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: 
nvshmem initialization failed, exiting 
nvshmem initialization failed, exiting 


W0305 11:55:57.242000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19119 via signal SIGTERM
W0305 11:55:57.244000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19120 via signal SIGTERM
W0305 11:55:57.245000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19121 via signal SIGTERM
W0305 11:55:57.245000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19122 via signal SIGTERM
W0305 11:55:57.245000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19124 via signal SIGTERM
W0305 11:55:57.246000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19125 via signal SIGTERM
W0305 11:55:57.246000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19126 via signal SIGTERM
Traceback (most recent call last):
  File "/work/DeepEP-main/tests/test_low_latency.py", line 164, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 204, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 4 terminated with exit code 255

By following the logs, we found the calltrace was test_loop() --> buffer = deep_ep.Buffer() --> __init__ --> self.runtime.sync(device_ids, ipc_handles, root_unique_id) in DeepEP and then nvshmemt_ibrc_connect_endpoints --> nvshmemt_ibrc_ep_connect --> ep_connect --> status = ftable.modify_qp(ep->qp, &attr, flags); in nvshmem_src_3.2.5-1. The status 110 is usually ETIMEDOUT。

[Baibaifan]

I have created a new branch that updates NVSHMEM to version 3.2.5. However, I don't have a RoCE environment for verification. Can you test this out? @Baibaifan Branch: https://github.com/deepseek-ai/DeepEP/tree/roce-support

I have the same problem on one 8*H100 cluster node for test_low_latency.py, and update NVSHMEM to version 3.2.5 with the newest patch. However, it doesn't work, it lacks for, with the same logs, i.e., /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 ibv_modify_qp failed ibv_modify_qp failed. However, I have tried the ib_write_bw cmd successfully. Any kind suggestion for this problem?

Could you provide the logs from your NVSHMEM 3.2.5 run? It appears you're still using NVSHMEM 3.1.7.

sorry for that old logs. The newest logs are as below, with some my comment logs.

(base) root@ubuntu:/work/DeepEP-main# python tests/test_low_latency.py 
local_rank:5, ip:127.0.0.1 port:3004
world_size:8, rank:5
local_rank:4, ip:127.0.0.1 port:3004
world_size:8, rank:4
local_rank:6, ip:127.0.0.1 port:3004
world_size:8, rank:6
local_rank:7, ip:127.0.0.1 port:3004
world_size:8, rank:7
local_rank:0, ip:127.0.0.1 port:3004
world_size:8, rank:0
local_rank:1, ip:127.0.0.1 port:3004
world_size:8, rank:1
local_rank:2, ip:127.0.0.1 port:3004
world_size:8, rank:2
local_rank:3, ip:127.0.0.1 port:3004
world_size:8, rank:3
setting....
setting....
setting....
setting....
setting....
setting....
setting....
setting....
setted...
rank:2, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
setted...
rank:0, num_ranks:8
Allocating buffer size: 2116.2912 MB ...
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
setted...
rank:1, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
setted...
rank:5, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
setted...
rank:4, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
after cpp.Buffer
setted...
rank:7, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
setted...
rank:3, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
after cpp.Buffer
setted...
rank:6, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 
ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 ep_connect failed 
transport create connect failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 
connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem setup connections failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 
ibv_modify_qp failed 
ibv_modify_qp failed 

ibv_modify_qp failed 
ibv_modify_qp failed 



/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed 
ep_connect failed 
ep_connect failed 

ep_connect failed 
ep_connect failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 

transport create connect failed 

transport create connect failed 
transport create connect failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 
connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 


connect EPS failed 

connect EPS failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 nvshmem setup connections failed 
nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 nvshmem setup connections failed 


nvshmem setup connections failed 

nvshmem setup connections failed 



/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 
ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed 
ep_connect failed 
ep_connect failed 

ep_connect failed 



/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 ep_connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 

transport create connect failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 transport create connect failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 ibv_modify_qp failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 ibv_modify_qp failed 
ibv_modify_qp failed 
connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 transport create connect failed 

connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 


connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
ep_connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 ep_connect failed 

ep_connect failed 
nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem setup connections failed 
connect EPS failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 

nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: transport create connect failed 

transport create connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 transport create connect failed 

nvshmem initialization failed, exiting 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: 
nvshmem initialization failed, exiting 
nvshmem setup connections failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 nvshmem initialization failed, exiting 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 nvshmem initialization failed, exiting 
connect EPS failed 


connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: connect EPS failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem initialization failed, exiting 

nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem setup connections failed 
nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: 

nvshmem initialization failed, exiting 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: 
nvshmem initialization failed, exiting 
nvshmem initialization failed, exiting 


W0305 11:55:57.242000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19119 via signal SIGTERM
W0305 11:55:57.244000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19120 via signal SIGTERM
W0305 11:55:57.245000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19121 via signal SIGTERM
W0305 11:55:57.245000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19122 via signal SIGTERM
W0305 11:55:57.245000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19124 via signal SIGTERM
W0305 11:55:57.246000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19125 via signal SIGTERM
W0305 11:55:57.246000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19126 via signal SIGTERM
Traceback (most recent call last):
  File "/work/DeepEP-main/tests/test_low_latency.py", line 164, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 204, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 4 terminated with exit code 255

By following the logs, we found the calltrace was test_loop() --> buffer = deep_ep.Buffer() --> __init__ --> self.runtime.sync(device_ids, ipc_handles, root_unique_id) in DeepEP and then nvshmemt_ibrc_connect_endpoints --> nvshmemt_ibrc_ep_connect --> ep_connect --> status = ftable.modify_qp(ep->qp, &attr, flags); in nvshmem_src_3.2.5-1. The status 110 is usually ETIMEDOUT。

I also encountered this problem, but tests/test_internode.py can be run, but the performance is worse. Is it related to the fact that I have 4 NICs per machine? @sphish

[kunfupanda-hw]

Hi @sphish, The process works, but the performance does not seem to meet expectations.

env:

1. H100 80GB HBM3 *8/HPC
2. 4 * CX7 NICs 400gb/s/HPC
3. RoCE

rank0 NCCL_DEBUG=INFO MASTER_ADDR=xxx WORLD_SIZE=2 RANK=0 python tests/test_internode.py

rank1 NCCL_DEBUG=INFO MASTER_ADDR=xxx WORLD_SIZE=2 RANK=1 python tests/test_internode.py

[tuning] Best dispatch (FP8): SMs 24, NVL chunk 28, RDMA chunk 8: 4.09 GB/s (RDMA), 13.36 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 28, RDMA chunk 32: 4.57 GB/s (RDMA), 14.91 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 3, RDMA chunk 32: 2.97 GB/s (RDMA), 9.69 GB/s (NVL)

The difference between the above speed and the official data is greater than the difference between the network cards. How can I test the performance with perftest? Is there any reference?

Hi, @Baibaifan, are MASTER_ADDRs of rank0/1 same? Are they IP of CX7 cards or host IP？

[Baibaifan]

Hi @sphish, The process works, but the performance does not seem to meet expectations.
env:

1. H100 80GB HBM3 *8/HPC
2. 4 * CX7 NICs 400gb/s/HPC
3. RoCE

rank0 NCCL_DEBUG=INFO MASTER_ADDR=xxx WORLD_SIZE=2 RANK=0 python tests/test_internode.py
rank1 NCCL_DEBUG=INFO MASTER_ADDR=xxx WORLD_SIZE=2 RANK=1 python tests/test_internode.py

[tuning] Best dispatch (FP8): SMs 24, NVL chunk 28, RDMA chunk 8: 4.09 GB/s (RDMA), 13.36 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 28, RDMA chunk 32: 4.57 GB/s (RDMA), 14.91 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 3, RDMA chunk 32: 2.97 GB/s (RDMA), 9.69 GB/s (NVL)

The difference between the above speed and the official data is greater than the difference between the network cards. How can I test the performance with perftest? Is there any reference?

Hi, @Baibaifan, are MASTER_ADDRs of rank0/1 same? Are they IP of CX7 cards or host IP？

MASTER_ADDRs are the same, both are host ip. It is to build Bootstrap use.

[sphish]

I have created a new branch that updates NVSHMEM to version 3.2.5. However, I don't have a RoCE environment for verification. Can you test this out? @Baibaifan Branch: https://github.com/deepseek-ai/DeepEP/tree/roce-support我已经创建了一个新分支，更新了 NVSHMEM 到版本 3.2.5。然而，我没有 RoCE 环境进行验证。你能测试一下吗？@Baibaifan 分支：https://github.com/deepseek-ai/DeepEP/tree/roce-support

I have the same problem on one 8*H100 cluster node for test_low_latency.py, and update NVSHMEM to version 3.2.5 with the newest patch. However, it doesn't work, it lacks for, with the same logs, i.e., /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 ibv_modify_qp failed ibv_modify_qp failed. However, I have tried the ib_write_bw cmd successfully. Any kind suggestion for this problem?我在一个 8 台 H100 集群节点（ test_low_latency.py ）上遇到了相同的问题，并将 NVSHMEM 更新到版本 3.2.5 并应用了最新的补丁。然而这并没有解决问题，日志依然相同，即 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 ibv_modify_qp failed ibv_modify_qp failed 。不过， ib_write_bw 命令我已经成功尝试过了。对于这个问题，有什么建议吗？

Could you provide the logs from your NVSHMEM 3.2.5 run? It appears you're still using NVSHMEM 3.1.7.您能提供 NVSHMEM 3.2.5 运行的日志吗？看来您仍在使用 NVSHMEM 3.1.7。

sorry for that old logs. The newest logs are as below, with some my comment logs.抱歉，旧的日志如下，以下是最新的日志，并附有一些我的注释。

(base) root@ubuntu:/work/DeepEP-main# python tests/test_low_latency.py 
local_rank:5, ip:127.0.0.1 port:3004
world_size:8, rank:5
local_rank:4, ip:127.0.0.1 port:3004
world_size:8, rank:4
local_rank:6, ip:127.0.0.1 port:3004
world_size:8, rank:6
local_rank:7, ip:127.0.0.1 port:3004
world_size:8, rank:7
local_rank:0, ip:127.0.0.1 port:3004
world_size:8, rank:0
local_rank:1, ip:127.0.0.1 port:3004
world_size:8, rank:1
local_rank:2, ip:127.0.0.1 port:3004
world_size:8, rank:2
local_rank:3, ip:127.0.0.1 port:3004
world_size:8, rank:3
setting....
setting....
setting....
setting....
setting....
setting....
setting....
setting....
setted...
rank:2, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
setted...
rank:0, num_ranks:8
Allocating buffer size: 2116.2912 MB ...
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
setted...
rank:1, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
setted...
rank:5, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
setted...
rank:4, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
after cpp.Buffer
setted...
rank:7, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
setted...
rank:3, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
after cpp.Buffer
setted...
rank:6, num_ranks:8
before init buffer
buffer.__init__
before cpp.Buffer
after cpp.Buffer
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
finish all_gather object
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
before runtime.sync
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 
ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 ep_connect failed 
transport create connect failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 
connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem setup connections failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 
ibv_modify_qp failed 
ibv_modify_qp failed 

ibv_modify_qp failed 
ibv_modify_qp failed 



/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed 
ep_connect failed 
ep_connect failed 

ep_connect failed 
ep_connect failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 

transport create connect failed 

transport create connect failed 
transport create connect failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 
connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 


connect EPS failed 

connect EPS failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 nvshmem setup connections failed 
nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 nvshmem setup connections failed 


nvshmem setup connections failed 

nvshmem setup connections failed 



/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 
ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 ibv_modify_qp failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ibv_modify_qp failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed 
ep_connect failed 
ep_connect failed 

ep_connect failed 



/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 ep_connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed 

transport create connect failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 transport create connect failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 110 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 ibv_modify_qp failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 ibv_modify_qp failed 
ibv_modify_qp failed 
connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 transport create connect failed 

connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 


connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
ep_connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 ep_connect failed 

ep_connect failed 
nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem setup connections failed 
connect EPS failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 

nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: transport create connect failed 

transport create connect failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 transport create connect failed 

nvshmem initialization failed, exiting 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: 
nvshmem initialization failed, exiting 
nvshmem setup connections failed 


/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 nvshmem initialization failed, exiting 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 nvshmem initialization failed, exiting 
connect EPS failed 


connect EPS failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: connect EPS failed 

/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem initialization failed, exiting 

nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 
nvshmem setup connections failed 
nvshmem setup connections failed 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: 

nvshmem initialization failed, exiting 
/work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: /work/nvshmem_src_3.2.5-1/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: 
nvshmem initialization failed, exiting 
nvshmem initialization failed, exiting 


W0305 11:55:57.242000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19119 via signal SIGTERM
W0305 11:55:57.244000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19120 via signal SIGTERM
W0305 11:55:57.245000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19121 via signal SIGTERM
W0305 11:55:57.245000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19122 via signal SIGTERM
W0305 11:55:57.245000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19124 via signal SIGTERM
W0305 11:55:57.246000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19125 via signal SIGTERM
W0305 11:55:57.246000 19054 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 19126 via signal SIGTERM
Traceback (most recent call last):
  File "/work/DeepEP-main/tests/test_low_latency.py", line 164, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 204, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 4 terminated with exit code 255

By following the logs, we found the calltrace was test_loop() --> buffer = deep_ep.Buffer() --> __init__ --> self.runtime.sync(device_ids, ipc_handles, root_unique_id) in DeepEP and then nvshmemt_ibrc_connect_endpoints --> nvshmemt_ibrc_ep_connect --> ep_connect --> status = ftable.modify_qp(ep->qp, &attr, flags); in nvshmem_src_3.2.5-1. The status 110 is usually ETIMEDOUT。通过查看日志，我们发现调用跟踪在 DeepEP 中为 test_loop() --> buffer = deep_ep.Buffer() --> __init__ --> self.runtime.sync(device_ids, ipc_handles, root_unique_id) ，然后在 nvshmem_src_3.2.5-1 中为 nvshmemt_ibrc_connect_endpoints --> nvshmemt_ibrc_ep_connect --> ep_connect --> status = ftable.modify_qp(ep->qp, &attr, flags); 。状态 110 通常为 ETIMEDOUT 。

Could you add the environment variable NVSHMEM_DEBUG=TRACE, run it again, and provide the log？