[KV Connector][Mooncake] Answer every pull and free dropped transfers promptly

[aoshen02]

Purpose

With prefill/decode disaggregation over MooncakeConnector, an abort on P could wedge D. We reproduced this on GB300 with 1P1D over RDMA under load: pause_generation(mode="abort") on P, or clients disconnecting, left D with KV fetches stuck for more than 10 minutes, and later requests timed out.

Root causes on P:

• Silent abort. When P aborted a request, nothing woke the pulls waiting on it. Each waited VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT (480 s), and a waiter that then found its state gone was dropped without a reply.
• Head-of-line blocking. P served pulls with 2 × num_workers fixed tasks, and a pull waiting for its prefill held one of them. Twenty aborted transfers blocked every later pull.
• KeyError. record_send_reqs popped the transfer of a request aborted before scheduling, with no default.

This PR guarantees two things:

• Every pull gets exactly one prompt answer.
• Each side's blocks are freed only when no transfer touches them.

P send state is explicit. SendState is one of:

• WAITING: P is still running the request, so pulls wait.
• READY: P holds the blocks for D.
• DONE: nothing more will be sent.

_end_send is the only place that ends a send. It frees P's blocks once no write reads them.

• A transfer that P aborts, that D releases, or that expires answers its pulls at once.
• When every D rank has pulled the KV, the state is dropped.
• A transfer that ended any other way leaves a tombstone, so a late pull also fails fast.
• A state expires one timeout after P is done with the request. While the request runs, it does not expire.

Pull admission. Each pull gets its own task, so a pull waiting for its prefill no longer delays the others.

• A pull holds blocks only once one of num_workers senders is free. Until then it can still expire or be aborted.
• Pulls whose transfer already ended are answered before any write in their batch.
• Each pull has a deadline: the abort timeout, counted from when P receives it. D gives up only 60 s later. The deadline bounds the wait for P and for a sender, and P checks it again after taking a sender. So no write starts after D could have given up.

Failed sends end the transfer at once, since D fails the load anyway. Before, they held P's blocks until expiry.

D side.

• Release: when a request finishes while its KV is still arriving, D asks P to drop the transfer, reusing the empty pull. D frees its blocks after P answers the original pull.
• Completion: a load ends only after every producer worker has answered, so one failing P worker cannot free blocks that another one still writes.

Stacked on vllm-project#59625 (Mooncake sleep mode); this fork PR targets its rebased branch.

Test Plan

• Unit tests:
  • test_send_state_answers_every_pull_once drives 18 orders of claim, ready, pull, abort, release, expiry, abort-all, a failing send, a write in flight, a busy sender pool, a deadline and a mixed batch. Each pull must be answered exactly once, and P's blocks must be freed exactly when nothing reads them.
  • test_load_ends_once_every_producer_answered covers the producer fan-out.
• Mutation testing: removing each key line fails the matching test. This covers the wake-up on end, claim and unclaimed expiry, failed-send cleanup, the hold during a write, sender admission and its deadline, the early abort reply, and the fan-out count.
• Hardware: GB300, Qwen3-8B, 1P1D over Mooncake RDMA across nodes. Each case:
  • runs 600 requests at concurrency 96, with 10% client disconnects;
  • applies a control call at t = 3 s;
  • then checks that the load finishes, both engines go idle, and greedy probes match.

Test Result

Case	before (main + vllm-project#59625)	this PR
pause abort on P (D under load)	D wedged	pass
pause abort on P and D	D pause 480 s	pass
sleep L1 abort on P and D	D sleep 480 s	pass
abort_requests on P	D wedged	pass
50% client disconnects	D wedged (96 stuck, on main)	pass

The lifecycle cases that clear the cache are fixed in the follow-up PR. The full 27-case matrix passes with both PRs.

Python ≥ 3.10.9 is assumed, for the asyncio semaphore cancellation fix (gh-90155).

🤖 Generated with Claude Code