[Fix][MoRI] Add MoRI-IO connector support

[simondanielsson]

Purpose

Fixes #126 and vllm-project/vllm#38692.

This PR adds

• --kv-connector moriio type for running vllm-router with MoRI-IO KV connector
• compatibility with MoRI connector with vllm 0.18.0+.
  • This PR introduces a transfer_id in the kv_transfer_params in the MoRI-IO connector and toy proxy (this PR is part of vllm 0.18.0). This PR applies the same for update the vllm-router, and also adds the required remote_dp_size for performing remote handshakes inside the MoRI connector. This allows us to support the MoRI-IO KV connector in vllm-router.

Important: This does not interfere with any other connectors functionality.

Basic usage

vllm-router \
    --vllm-pd-disaggregation \
    --kv-connector moriio \
    --vllm-discovery-address "0.0.0.0:${PROXY_PING_PORT}" \
    --port "${ROUTER_PORT}" \
    --host 0.0.0.0 \

Dependencies

• Hard dependency: (vllm) to slightly update connector messages to align with the P2pNcclConnector: [Fix][MoRI] Align MoRI-IO message format with P2pNcclConnector and vllm-router vllm#39565
• Nice to have (can be merged whenever): (router) add streaming support in the service discovery path in order to run vllm bench serve with the vllm backend: feat: support stream response in the process_vllm_two_stage_request_discovered #114
  Order of merging should be (1) the vllm PR (2) this PR (3) the streaming PR

Test Plan

This PR can be used in conjunction with this vllm PR to run vllm w/ MoRI-IO KV connector and vllm-router.

Reproducer scripts can be found in this temporary branch: mpashkovskii/vllm#4

It includes

1. Dockerfiles for building the branches from source (incl installation of Broadcom NIC drivers, etc.),
2. Scripts for testing MoRI+vllm-router using the images built in point 1, both in single-node (run_pd_demo.sh) and multi-node setups (run_pd_demo_2node.sh‎)

The tests include both basic smoke test (example request), vllm bench serve, and GSM8K using lm_eval (although the latter two require streaming support as added in the PR(s) mentioned above).

Test Result

See vllm-project/vllm#39565 for full performance and accuracy benchmarks. Below we show results from minimal smoke test reproducer.

2 node using DSR1

Basic smoke test (note random params):

$ IS_PREFILL=1 PREFILL_IP=10.21.9.47 DECODE_IP=10.21.9.29     ./examples/online_serving/disaggregated_serving/moriio_pd_demo/run_pd_demo_2node.sh



Containers will be shut down automatically when this script exits.
(Set KEEP_ALIVE=1 to leave them running.)

>>> Smoke test: sending a completion request through vllm-router...
2026-04-10 15:06:06  INFO vllm_router_rs::routers::http::vllm_pd_router: src/routers/http/vllm_pd_router.rs:1436: vLLM route_completion called, use_discovery=true
2026-04-10 15:06:06  INFO vllm_router_rs::routers::http::vllm_pd_router: src/routers/http/vllm_pd_router.rs:1443: Using service discovery mode, processing vLLM two-stage request
2026-04-10 15:06:06  INFO vllm_router_rs::policies::consistent_hash: src/policies/consistent_hash.rs:281: Updated consistent hash ring with 1 workers and 160 virtual nodes
2026-04-10 15:06:06  INFO vllm_router_rs::policies::consistent_hash: src/policies/consistent_hash.rs:593: CONSISTENT_HASH_DEBUG: Extracted hash key: request_hash:f9454da6e4bbbd0a
2026-04-10 15:06:06  INFO vllm_router_rs::policies::consistent_hash: src/policies/consistent_hash.rs:598: CONSISTENT_HASH_DEBUG: Hash key 'request_hash:f9454da6e4bbbd0a' mapped to worker: http://10.21.9.47:8100
2026-04-10 15:06:06  INFO vllm_router_rs::policies::consistent_hash: src/policies/consistent_hash.rs:647: Consistent hash routing: key='request_hash:f9454da6e4bbbd0a' -> worker='http://10.21.9.47:8100' (index=0)
2026-04-10 15:06:06  INFO vllm_router_rs::policies::consistent_hash: src/policies/consistent_hash.rs:281: Updated consistent hash ring with 1 workers and 160 virtual nodes
2026-04-10 15:06:06  INFO vllm_router_rs::policies::consistent_hash: src/policies/consistent_hash.rs:593: CONSISTENT_HASH_DEBUG: Extracted hash key: request_hash:f9454da6e4bbbd0a
2026-04-10 15:06:06  INFO vllm_router_rs::policies::consistent_hash: src/policies/consistent_hash.rs:598: CONSISTENT_HASH_DEBUG: Hash key 'request_hash:f9454da6e4bbbd0a' mapped to worker: http://10.21.9.29:8200
2026-04-10 15:06:06  INFO vllm_router_rs::policies::consistent_hash: src/policies/consistent_hash.rs:647: Consistent hash routing: key='request_hash:f9454da6e4bbbd0a' -> worker='http://10.21.9.29:8200' (index=0)
[aiter] pciChipId: 0x74a1, CU count: 304
[aiter] hipModuleLoad: /usr/local/lib/python3.12/dist-packages/aiter_meta/hsa//gfx942/fmha_v3_fwd/MI300/fwd_hd192x128_bf16_causal_rtna_group.co GetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE
[aiter] hipModuleGetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE Success
[aiter] pciChipId: 0x74a1, CU count: 304
[aiter] hipModuleLoad: /usr/local/lib/python3.12/dist-packages/aiter_meta/hsa//gfx942/fmha_v3_fwd/MI300/fwd_hd192x128_bf16_causal_rtna_group.co GetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE
[aiter] pciChipId: 0x74a1, CU count: 304
[aiter] hipModuleLoad: /usr/local/lib/python3.12/dist-packages/aiter_meta/hsa//gfx942/fmha_v3_fwd/MI300/fwd_hd192x128_bf16_causal_rtna_group.co GetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE
[aiter] pciChipId: 0x74a1, CU count: 304
[aiter] hipModuleLoad: /usr/local/lib/python3.12/dist-packages/aiter_meta/hsa//gfx942/fmha_v3_fwd/MI300/fwd_hd192x128_bf16_causal_rtna_group.co GetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE
[aiter] hipModuleGetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE Success
[aiter] hipModuleGetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE Success
[aiter] hipModuleGetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE Success
[aiter] pciChipId: 0x74a1, CU count: 304
[aiter] hipModuleLoad: /usr/local/lib/python3.12/dist-packages/aiter_meta/hsa//gfx942/fmha_v3_fwd/MI300/fwd_hd192x128_bf16_causal_rtna_group.co GetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE
[aiter] pciChipId: 0x74a1, CU count: 304
[aiter] hipModuleLoad: /usr/local/lib/python3.12/dist-packages/aiter_meta/hsa//gfx942/fmha_v3_fwd/MI300/fwd_hd192x128_bf16_causal_rtna_group.co GetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE
[aiter] hipModuleGetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE Success
[aiter] pciChipId: 0x74a1, CU count: 304
[aiter] hipModuleLoad: /usr/local/lib/python3.12/dist-packages/aiter_meta/hsa//gfx942/fmha_v3_fwd/MI300/fwd_hd192x128_bf16_causal_rtna_group.co GetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE
[aiter] hipModuleGetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE Success
[aiter] hipModuleGetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE Success
[aiter] pciChipId: 0x74a1, CU count: 304
[aiter] hipModuleLoad: /usr/local/lib/python3.12/dist-packages/aiter_meta/hsa//gfx942/fmha_v3_fwd/MI300/fwd_hd192x128_bf16_causal_rtna_group.co GetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE
[aiter] hipModuleGetFunction: _ZN5aiter41fmha_fwd_hd192x128_bf16_causal_rtna_groupE Success
(APIServer pid=7) INFO:     10.21.9.47:58400 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=7) INFO 04-10 15:07:08 [loggers.py:259] Engine 000: Avg prompt throughput: 0.5 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 0.0%
(APIServer pid=7) INFO 04-10 15:07:18 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 0.0%
{
    "id": "cmpl-___prefill_addr_host:10.21.9.47,handshake:6301,notify:61005___decode_addr_host:10.21.9.29,handshake:6301,notify:61005_ab377ab0c004415e8a74fbc0e7e1fe62",
    "object": "text_completion",
    "created": 1775833620,
    "model": "deepseek-ai/DeepSeek-R1-0528",
    "choices": [
        {
            "index": 0,
            "text": ".B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42.B\u8bf7\u6c42",
            "logprobs": null,
            "finish_reason": "length",
            "stop_reason": null,
            "token_ids": null,
            "prompt_logprobs": null,
            "prompt_token_ids": null
        }
    ],
    "service_tier": null,
    "system_fingerprint": null,
    "usage": {
        "prompt_tokens": 5,
        "total_tokens": 69,
        "completion_tokens": 64,
        "prompt_tokens_details": null
    },
    "kv_transfer_params": null
}

(Set USE_BENCH=1 for a perf benchmark, or USE_GSM8K=1 for accuracy eval.)

>>> Shutting down containers...
moriio-prefill
moriio-router
Done.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results

[simondanielsson]

remote_host and remote_port are not read by any prefill instance's connector, so it's safe to remove them

[simon-mo]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Bravo.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[BugenZhao]

Rest LGTM. Thanks!

[simondanielsson]

@BugenZhao Does this failing CI look like an unrelated issue to you as well? 🙏

[BugenZhao]

@BugenZhao Does this failing CI look like an unrelated issue to you as well? 🙏

Yes I think it's #154 aims to resolve.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 328e9ce50f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Accept moriio in Python kv_connector validation

Adding KvConnector::MoriIO here does not make it usable from the Python API, because Router::to_config still only accepts "nixl" and "mooncake" and returns ValidationFailed for any other value (src/lib.rs, kv_connector match). In environments configuring the router through Python bindings, kv_connector="moriio" now fails at startup even though the enum advertises support.

Useful? React with 👍 / 👎.