[Mooncake] Auto-discover compatible RDMA devices for Mooncake Connector

[wzhao18]

Purpose

The PR supports auto-discovery for compatible and mutually routable RDMA devices for use by the Mooncake connectors (e.g., MooncakeConnector, MooncakeStoreConnector) when no device names are specified. The existing device selection in mooncake can use all devices in /sys/class/infiniband as candidates, which can lead to errors:

• devices that have node_type=7: unspecified and link_layer=Unknown are not usable
• devices with limited membership (pkey0=0x7fff) can cause transfer failure, despite successful startup.
• some devices are not mutually routable - leading to certain transfer to succeed while others fail

Example (B300, 16 RNICs):

> ls /sys/class/infiniband
mlx5_0  mlx5_1  mlx5_10  mlx5_11  mlx5_12  mlx5_13  mlx5_14  mlx5_15  mlx5_16  mlx5_17  mlx5_20  mlx5_21  mlx5_22  mlx5_23  mlx5_6  mlx5_7

HCA      ROUTE_GROUP
mlx5_0   172.16.0.0/13
mlx5_6   172.16.0.0/13
mlx5_10  172.16.0.0/13
mlx5_12  172.16.0.0/13
mlx5_14  172.16.0.0/13
mlx5_16  172.16.0.0/13
mlx5_20  172.16.0.0/13
mlx5_22  172.16.0.0/13

mlx5_1   172.24.0.0/13
mlx5_7   172.24.0.0/13
mlx5_11  172.24.0.0/13
mlx5_13  172.24.0.0/13
mlx5_15  172.24.0.0/13
mlx5_17  172.24.0.0/13
mlx5_21  172.24.0.0/13
mlx5_23  172.24.0.0/13

Same rail ping (success):

> bash -lc 'timeout 12s ibv_rc_pingpong -d mlx5_0 -g 3 -n 10 -s 4096 -p 18611 >/tmp/rc_same_server.log 2>&1 & spid=$!; sleep 1; timeout 12s ibv_rc_pingpong -d mlx5_6 -g 3 -n 10 -s 4096 -p 18611 172.20.0.20; client_rc=$?; wait $spid; server_rc=$?; echo "client_rc=$client_rc server_rc=$server_rc"; cat /tmp/rc_same_server.log'
  local address:  LID 0x0000, QPN 0x004384, PSN 0xb3adf5, GID ::ffff:172.22.0.20
  remote address: LID 0x0000, QPN 0x0046f6, PSN 0x8d5048, GID ::ffff:172.20.0.20
81920 bytes in 0.00 seconds = 3343.67 Mbit/sec
10 iters in 0.00 seconds = 19.60 usec/iter
client_rc=0 server_rc=0
  local address:  LID 0x0000, QPN 0x0046f6, PSN 0x8d5048, GID ::ffff:172.20.0.20
  remote address: LID 0x0000, QPN 0x004384, PSN 0xb3adf5, GID ::ffff:172.22.0.20
81920 bytes in 0.00 seconds = 220.36 Mbit/sec
10 iters in 0.00 seconds = 297.40 usec/iter

Cross-rail ping (fail):

> bash -lc 'timeout 12s ibv_rc_pingpong -d mlx5_0 -g 3 -n 10 -s 4096 -p 18612 >/tmp/rc_cross_server.log 2>&1 & spid=$!; sleep 1; timeout 12s ibv_rc_pingpong -d mlx5_1 -g 3 -n 10 -s 4096 -p 18612 172.20.0.20; client_rc=$?; wait $spid; server_rc=$?; echo "client_rc=$client_rc server_rc=$server_rc"; cat /tmp/rc_cross_server.log'
  local address:  LID 0x0000, QPN 0x003d55, PSN 0x146428, GID ::ffff:172.28.0.20
client read/write: No space left on device
Couldn't read/write remote address
client_rc=1 server_rc=1
Failed to modify QP to RTR
Couldn't connect to remote QP
  local address:  LID 0x0000, QPN 0x0046f7, PSN 0x04d28e, GID ::ffff:172.20.0.20

Test Plan

Test Result

• Tested with llama8b and deepseekv4 single node and multinode disagg with KV offloading:

  PREFILL_CFG='{
    "kv_connector": "MultiConnector",
    "kv_role": "kv_producer",
    "kv_connector_extra_config": {
      "connectors": [
        {"kv_connector": "MooncakeConnector", "kv_role": "kv_producer"},
        {"kv_connector": "MooncakeStoreConnector", "kv_role": "kv_producer"}
      ]
    }
  }'

VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve deepseek-ai/DeepSeek-V4-Pro \
  --kv-cache-dtype fp8 \
  --trust-remote-code \
  --block-size 256 \
  --enable-prefix-caching \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --attention_config.use_fp4_indexer_cache True \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --max-cudagraph-capture-size 2048 \
  --max-num-batched-tokens 2048 \
  --no-enable-flashinfer-autotune \
  --tensor-parallel-size 8 \
  --host 0.0.0.0 \
  --port 8100 \
  --kv-transfer-config "$PREFILL_CFG"

DECODE_CFG='{
    "kv_connector": "MultiConnector",
    "kv_role": "kv_consumer",
    "kv_connector_extra_config": {
      "connectors": [
        {"kv_connector": "MooncakeConnector", "kv_role": "kv_consumer"},
        {"kv_connector": "MooncakeStoreConnector", "kv_role": "kv_consumer"}
      ]
    }
  }'

VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve deepseek-ai/DeepSeek-V4-Pro \
  --kv-cache-dtype fp8 \
  --trust-remote-code \
  --block-size 256 \
  --enable-prefix-caching \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --attention_config.use_fp4_indexer_cache True \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --max-cudagraph-capture-size 2048 \
  --max-num-batched-tokens 2048 \
  --no-enable-flashinfer-autotune \
  --tensor-parallel-size 8 \
  --host 0.0.0.0 \
  --port 8200 \
  --kv-transfer-config "$DECODE_CFG"

  python examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py \
    --host 0.0.0.0 --port 8000 \
    --prefill http://$PREFILL_IP:8100 50052 \
    --decode  http://$DECODE_IP:8200

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wzhao18.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork