Repository navigation
[KV Connector] Add ConnexConnector: P/D disaggregation that survives decode-instance failures - #63804
yanyinglin wants to merge 1 commit into
Conversation
…nt mobility Register ConnexConnector, a pull-based P/D connector implemented in the connex-ai package (optional dependency, imported lazily). The decode instance reads prompt KV with one-sided RDMA reads (connex-rs over UCX with GPUDirect, or Mooncake Transfer Engine); the prefill instance keeps the blocks as a renewable hold until the decode instance releases them, so a request whose decode instance fails can be resumed elsewhere without a second prefill. Adds the registration, a usage page, the disagg_prefill.md entry and scheduler-level tests (skipped without connex-ai). Signed-off-by: yanyinglin <linyanying19@mails.ucas.ac.cn>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
Documentation preview: https://vllm--63804.org.readthedocs.build/en/63804/ |
mergify
Bot
added
documentation
There was a problem hiding this comment.
🟡 Changes recommended
The delegated adapter can report receive completion for a running cached-prefix request, triggering a scheduler assertion during resume.
4 open issues
What changed in this PR
Adds optional ConnexConnector integration for prefill/decode disaggregation, using retained prefill KV to support recovery from decode-instance failures without changing other connectors.
Changes:
- Registers the connector through a lazy wrapper around
connex-ai, with installation guidance. - Adds scheduler-level tests for KV retention, asynchronous loading, and release.
- Documents setup, configuration, and supported deployment constraints.
| File | Description |
|---|---|
vllm/distributed/kv_transfer/kv_connector/v1/connex_connector.py |
Re-exports the optional connector with an installation error message. |
vllm/distributed/kv_transfer/kv_connector/factory.py |
Registers ConnexConnector for lazy loading. |
tests/v1/kv_connector/unit/test_connex_connector.py |
Tests scheduler handoff and release when connex-ai is installed. |
docs/features/disagg_prefill.md |
Adds Connex to the connector list. |
docs/features/connex_connector_usage.md |
Documents installation, deployment, configuration, and limitations. |
Review effort: Balanced
|
|
||
| try: | ||
| from connex.adapters.vllm.pd_connector import ( | ||
| ConnexConnector, |
|
|
||
| | key | default | meaning | | ||
| | --- | --- | --- | | ||
| | `backend` | `auto` | `connex-rs`, `mooncake` or `auto` | |
| until the decode instance releases them (or a lease expires), so if the | ||
| decode instance fails mid-generation, the request can be resumed on another | ||
| decode instance that pulls the same KV again instead of re-running prefill. |
| Decode instance: | ||
|
|
||
| ```bash | ||
| vllm serve meta-llama/Meta-Llama-3-8B --port 8200 \ |
Overview
Registers
ConnexConnector, a pull-based P/D connector whose prefill-side KV stays available as a renewable hold, so a request whose decode instance dies can be resumed on another decode instance without a second prefill. The implementation ships in the optionalconnex-aipackage (same pattern asLMCacheConnectorV1); vLLM gains no dependency and no code path changes for other connectors.Not a duplicate: searched open PRs/issues for
connex, P/D fault tolerance and elastic P/D connectors. #55151 ("[wip] support PD seperate mode for fault-tolerance", no description or code yet) targets fault tolerance in vLLM's own P/D path; this PR does not touch that path and adds an external connector with request-level resume. #59330 (stop P/D transfers at N-1) changes NIXL/Mooncake semantics for sliding-window models; ConnexConnector currently supports full-attention models only and follows the current N-token load.Claims
kv_connector="ConnexConnector": decode pulls prompt KV with one-sided RDMA reads (connex-rs over UCX with GPUDirect RDMA, or Mooncake Transfer Engine).NixlConnectoron the same setup.kv_load_failure_policy).Validation
On this branch (main d0d681e) with current connex-ai:
test_connex_connector.py2 passed,test_config.py17 passed, pre-commit on changed files passes. connex-ai's own suite covers the connector and router (CPU, fake engines) and the transfer core.CI coverage gap: the new tests
importorskipconnex-ai, which is not installed in vLLM CI, so CI will skip them; the validation above was run outside CI.End-to-end runs were done on vLLM 0.32.0 (not re-run on d0d681e): 4x A40 on one host, ConnectX HDR, GPUDirect RDMA, Llama-3-8B, Connex PD router, prefill and decode on different GPUs.
Throughput/latency vs NixlConnector (1024-token prompts, 128 output tokens, same router and hosts)
Details
Existing P/D connectors free the prefill blocks as soon as decode has read them, so once decoding starts the prompt KV exists in one place. If the decode instance dies (crash, preemption, spot scale-down), every in-flight request must be prefilled again or errors out.
ConnexConnector makes the prefill-side copy a hold: a lease named by a transfer id, renewed by every pull, released by the decode instance when the request finishes or dropped after
retain_s(default 60 s).release: after_pullfrees blocks after the first read like other connectors. To resume, a router resends the samekv_transfer_paramswith the prompt plus the k generated tokens; the new decode instance async-loadsremote_num_tokensand computes the k tokens locally. Instances can register their P/D role with the Connex front-router so instances can be added or removed at run time.Request flow:
kv_producer) runs the prompt; on finish the scheduler keeps the blocks and returnstransfer_id,remote_ctrl,remote_num_tokens.kv_consumer) schedules the request asWAITING_FOR_REMOTE_KVSand pulls the blocks with one-sided reads.Changes:
v1/connex_connector.py(lazy re-export with install hint), registration infactory.py,docs/features/connex_connector_usage.mdplus an entry indisagg_prefill.md, and scheduler-level tests through the realScheduler.Limitations: full-attention KV only; equal TP and block size on both sides.
Developed with AI coding assistance; all changes were reviewed and tested by the author.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
Need help on this PR? Tag
@codesmith-botwith what you need.