[ShangmingCai]

Motivation

Extracts the one live bug fix from #38704 (@kflansburg) without the extra per-request gloo all-reduce or the post-enqueue abort honoring, which have no reachable failure on main. Thanks to Kevin for identifying the restore consensus problem; he is credited as co-author on the commit.

With --disaggregation-decode-enable-radix-cache --enable-hierarchical-cache on an attention-TP decode instance, two places branch on the rank-local HiCache restore status after the MIN-reduced KV poll:

1. DecodeTransferQueue.pop_transferred did if poll == KVPoll.Failed or hicache_restore_status == FAILED:. A rank whose L2->L1 load_back() fails (the production trigger is UnifiedRadixCache.load_back() returning False when device pages cannot be allocated) aborted the request and freed its pages alone, while its TP peers took the reduced poll (Success) and committed the same request. The transfer queues then differ in length and the next collective dies with gloo ... op.preamble.length <= op.nbytes. 2 vs 1, taking the whole decode server down. The same branch also fired while the P->D transfer was still in flight, releasing pages prefill was still writing.
2. _hicache_pending_restore_tokens excluded FAILED requests from the admission reservation, so the failed rank saw restore_token_count more allocatable tokens than the peer that had allocated the restore, admitted one request more, and the queues diverged the same way one tick later. [Disaggregation] Honor admission aborts before KV handoff #38704 did not cover this one, so its gate alone still crashes.

The PENDING half of (1) was already fixed by #41393 (HiCacheRestoreGatedKVReceiver), but poll_and_all_reduce_with_staging polled the raw kv_receiver and still relied on a rank-local PENDING continue in pop_transferred.

Modifications

• HiCacheRestoreGatedKVReceiver.poll: when the transport poll is Success, PENDING still demotes to Transferring; FAILED now returns Failed. A non-terminal transfer is never pre-empted, so the pages are never released while prefill may be writing them. The outcome rides the existing MIN all-reduce, so every rank fails the request in the same tick.
• pop_transferred branches on the reduced poll only. A local restore failure is reported as HiCache restore failed instead of "unknown reason from another rank". The rank-local PENDING continue is gone.
• poll_and_all_reduce_with_staging takes pollers=, and DecodeTransferQueue passes the gated pollers on both poll paths.
• _hicache_pending_restore_tokens keeps the reservation for every queued request whose restore has not been allocated on this rank (hicache_restore_lock is None), FAILED included, until the reduced poll retires it everywhere.
• Tests: TestDecodeHiCacheRestoreGate in test_decode_queue_cleanup.py (gate truth table, reservation, staging path uses the gated pollers, pop_transferred against a simulated peer through a MIN reduce). All four are red on main and green here.

No new collective. Net diff is +209/-38 over 4 files.

Reproduction and verification

1P (TP1) + 1D (TP2), Qwen3-0.6B, decode HiCache (--hicache-ratio 4 --hicache-write-policy write_through --max-total-tokens 24000), NIXL transfer, sglang_router --pd-disaggregation. 24 prompts of 3000 tokens replayed in rounds so the second round is served from L2 (load_back_tokens_total = 72000 per rank per round). A sitecustomize wrapper forces UnifiedRadixCache.load_back() to return False on TP1 for one request.

	main	this PR
TP1 log	Decode transfer failed on TP1 only	both ranks fail that one request, reason HiCache restore failed
next collective	gloo ... op.preamble.length <= op.nbytes. 2 vs 1, decode SIGQUIT	ok
round with the injected failure	24/24 timeouts, decode dead	24/24 ok (router retried the failed request)
following rounds	No available decode workers	24/24 ok, fully L2-restored

test/registered/unit/disaggregation/: 681 passed on this branch rebased on current main.

Accuracy Tests

N/A, no change to model forward.

Speed Tests and Profiling

N/A, no new collective; the gated poll is a field read on the existing per-tick path.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ✅ Run #38062799237
Latest PR Test (Extra): ❌ Run #38062799010
Latest PR Test (AMD ROCm 10): ❌ Run #38062799249

[ShangmingCai]

/tag-and-rerun-ci