[MoRIIO] Preserve hybrid READ tracking on failure and timeout

[whx-sjtu]

Overview

Hybrid READ completion cleanup can discard transfer tracking, or notify P to release blocks, while another transfer remains pending. Raise TransferError on failure or timeout before cleanup changes ownership.

Rebased onto main bfa7e5b7afe40ebe48288801a2baf5c1135cd881. This remains an independent child of closed #58968.

Claims

D observes hybrid READ [FAILED, PENDING] or an expired pending READ
- enter completion cleanup and lose in-flight tracking
+ raise before release notification or tracking deletion

Successful hybrid completion and existing attention-only recovery retain their behavior.

Validation

Validated on 7fac33a0118fba54e5f83f612e40088ece8ca8d3 (2026-10-12): 34 passed. Changed-file pre-commit passed, including mypy for Python 3.11 and test/CI tethering checks.

VLLM_TARGET_DEVICE=cpu .venv/bin/python -m pytest -q -ra \
  tests/v1/kv_connector/unit/test_moriio_tp_ack.py

The suite covers both failed/pending orders, timeout, completed transfers, a pending request at the deadline and attention-only failure. It asserts release notifications, transfer tracking and invalid-block reporting. Original October 8 negative controls (194da61de1) gave 3 failed / 3 passing controls, versus six passing cases with the fix.

No new model/RDMA run was performed. Exact-source fault-injection/teardown validation remains pending. Raising prevents this cleanup path from releasing ownership; it does not prove DMA drain or revocation.

The eight independent PRs compose on this same main. The combined CPU suite passes 344 tests, with 33 skips (32 ROCm/aiter-gated connector cases and one GPU LLM case) and one model-loading deselection. Counts overlap with the independent suites.

Details

The normal layer/step wait already raises on failed transfers. The missing guard is in _pop_done_transfers, reached from READ completion collection. Hybrid state has no safe fallback through the attention-only invalid-block recovery path.

Raising prevents this cleanup path from releasing ownership; it does not drain/revoke DMA or provide automatic recovery. Producer retention until a real ACK is separate work in #56465. That PR also modifies consumer timeout notification, so composition must preserve this hybrid exception before any timeout release. #58962 concerns WRITE consumer timeout; #60544 adds hybrid WRITE and batch waits. Neither is this READ cleanup guard. #58968 was closed after the split.

Merge danger: two-way door; blast radius: MoRIIO hybrid READ error handling. A failed hybrid transfer aborts instead of continuing with incomplete recurrent state.

#60808 moves input waits into start_load_kv; it does not provide DMA drain/revoke or replace this completion-cleanup guard. Producer ACK retention remains separate in #56465.

AI assistance was used for implementation, rebase, regression checks and this description. Human line-by-line review and the runtime acceptance items above remain pending. No performance improvement is claimed.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
• AI assistance was used during the creation of this PR.
• Design Fit: Source/compatibility review completed; human line-by-line review remains pending.
• Testing and Validation: Rebased CPU regressions passed; the runtime acceptance gaps above remain open.
• Code Quality and Style: Changed-file pre-commit, mypy and test/CI tethering checks passed.
• Pull Request Contents: Current source, evidence, related PRs and remaining limitations documented.