[Bugfix] Drain Mooncake transfers before releasing KV pages

[kflansburg]

Motivation

Mooncake aborts and transfer timeouts currently make requests terminal before in-flight native transfers have quiesced. The scheduler can therefore return source or destination KV pages to the allocator while an existing RDMA, executor, or CUDA staging operation still reads or writes those addresses. If a page is reassigned, the old transfer can corrupt another request or read its KV state.

The existing ABORT_ACK only confirms that the abort notification was received. It does not prove that synchronous native transfer work has returned.

Modifications

• Add a per-room transfer lifetime that closes admission on abort and tracks active Mooncake transfer workers.
• Drain all submitted executor futures, including running siblings after an error or partial submission failure, before releasing the room lifetime.
• Add tokenized, retryable abort notifications and defer ABORT_ACK until prefill transfer work is quiescent.
• Advertise and require the QUIESCE_V1 capability on both peers so mixed versions fail before transferring KV rather than silently weakening the ownership barrier.
• Propagate logical failure across TP and CP ranks first, then report terminal failure only after every participant confirms local transfer quiescence.
• Retain decode staging allocations and chunked-prefill KV ownership until RDMA and CUDA staging work is complete, while rejecting late staging and auxiliary messages for aborted rooms.
• Serialize messages sent through shared Mooncake ZMQ endpoints so status, auxiliary data, and abort acknowledgements preserve ordering.
• Add deterministic unit coverage for blocked native transfers, sibling executor drain, capability negotiation, token retries, collective failure barriers, staging cleanup races, message ordering, and chunked-prefill retention.

This protocol requires coordinated rollout of prefill and decode peers. A peer that does not advertise QUIESCE_V1 is rejected before KV transfer begins.

Accuracy Tests

This change does not modify model forward computation or model outputs. It prevents stale transfers from reading or overwriting KV pages after request failure.

Focused and adjacent CPU unit tests passed:

26 focused tests passed
104 focused-plus-adjacent tests passed

Speed Tests and Profiling

No performance benchmarks were run. The normal data-transfer path is unchanged apart from lifecycle accounting and per-endpoint send serialization. The additional waiting occurs only when a transfer is aborted, fails, or must drain before ownership release.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #29438714597
Latest PR Test (Extra): ❌ Run #29438714032

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[KastanDay]

I this this patch is better addressed by #32564

[github-actions]

Thanks @kflansburg. Closing this because it is still a draft and has not been updated in 40 days.

Reopen it if the work is still relevant.

Some directories moved recently, so an older branch may need retargeting:
sgl-kernel/ -> python/sglang/kernels/aot/, python/sglang/jit_kernel/
-> python/sglang/kernels/jit/, docs/ -> docs/docs/ (.mdx),
bench_serving.py -> benchmark/serving.py, test/srt/ -> test/registered/.