[v0.5.15 19/21] Retain KV ownership until Mooncake transfer quiescence

[KastanDay]

Motivation

Classification: Primary cross-request corruption fix

Abort and timeout cleanup previously freed pages while native transfer work still held pointers, allowing another request to reuse a page before the old operation resumed.

Modifications

Introduce room lifetimes, authenticated abort tokens, leases and tombstones, sibling-future joining, rank-wide drain and deferred release across Mooncake, NIXL, Ascend, staging, prefill and decode.

Touched scope: 11 disaggregation, staging, scheduler and connector files

This is patch 19/21 in the exact ordered SGLang v0.5.15 validation stack. The source base is f63458b5beaceabbd9d749b9fc956370e1b649e6. This PR targets the immediately preceding review branch, so the displayed diff is exactly one tested patch.

Experimental validation

Decisive red/green: A blocked transfer captured page 42; vulnerable cleanup freed it; B reused it and wrote request-b-canary; A resumed and read B data. Patched code withheld the page until TRANSFER_END and preserved A data. Nine focused tests and then 23 heavy lifecycle tests passed. Live 2P/1D and 1P/2D true AbortReq waves returned 8/8 correct victims with zero engine restarts.

Where applicable, the integrated staging control was an isolated GLM 5.2 deployment on three H200 nodes in both 2-prefill/1-decode and 1-prefill/2-decode topologies. Genuine stream=true disconnects reached active KV transfer and logged KVTransferError: Aborted by AbortReq; bounded victims were 8/8 correct, with no foreign canary and no engine restart. This integrated result is not presented as per-commit isolation.

Reproducible verification

Run TestFinding01TransferQuiescence with exact barriers, never sleeps, then repeat the documented stream=true eight-donor cutoff workload on an isolated 2P/1D deployment.

Common source checks for this PR:

git diff --check kv-18-omit-continuous-cache-details..kv-19-mooncake-transfer-quiescence
PYTHONPYCACHEPREFIX=/tmp/sglang-pyc python3 -m py_compile $(git diff --name-only kv-18-omit-continuous-cache-details..kv-19-mooncake-transfer-quiescence -- 'python/*.py' 'python/**/*.py')

The deterministic harness uses AST extraction of audited methods plus fake transport/kernel barriers. It requires Python with torch installed and deliberately uses acknowledged barriers instead of timing sleeps.

Accuracy and performance

No throughput improvement is claimed. Correctness patches fail closed or defer release until ownership is safe. Instrumentation and compatibility carries are labeled explicitly above.

Checklist

• Exact v0.5.15 patch reconstructed as one commit
• Python compilation and git diff --check
• Validation scope and limitations stated
• Run repository CI on the fork
• Forward-port to current main only after checking equivalent upstream fixes and semantic drift

[KastanDay]

Superseded by combined current-main PR #23. That replacement rebases Kevin Flansburg's upstream quiescence draft sgl-project#31357 and integrates the required bounded early-metadata rendezvous from fork PR #20. This old v0.5.15 stack branch is retained for history.