[PD][NIXL] Prevent abort ACK with unresolved errored transfers

[200lz]

Motivation

#35360 made _staging_outstanding the Prefill-side drain condition for
deferred Decode-side KV release: chunks are counted at dequeue so that
Failed with outstanding == 0 means there is no remaining transfer work
that can still write.

The follow-up fix in 30a50a2 relies on the same invariant for NIXL errors:
when the NIXL DONE barrier exits on ERR, the transfer group remains counted,
so ABORT_ACK is withheld and Decode falls back to the release timeout rather
than treating the room as drained.

The current Failed-room skip path can defeat that invariant.

If another chunk for the same room was already queued, it is dequeued after
the room becomes Failed. The worker first counts that chunk, then the skip
path removes the entire room-level _staging_outstanding entry with pop().
That also erases the deliberately retained count from the earlier errored
transfer group.

A later deferred-ACK check can therefore observe outstanding == 0 and emit
ABORT_ACK even though the earlier NIXL transfer group has not been proven
quiescent.

A deterministic negative control reproduces this on upstream main:

existing unresolved count: 1
stale Failed-room chunk dequeued: 1 -> 2
current skip path: pop(room)
observed count: 0
ABORT_ACK emitted: ('10.0.0.8', 6000, 73)

This PR restores the intended outstanding-count invariant.

Changes

The PR is intentionally split into two independently reviewable commits.

1. Preserve unresolved accounting on the Failed-room skip

fix(pd): preserve unresolved count when skipping failed NIXL chunks

The Failed-room skip now removes only the skipped chunk's own contribution
instead of clearing the entire room-level count.

This preserves any earlier unresolved group's outstanding state and keeps
ABORT_ACK blocked.

2. Retain unresolved handles after NIXL transfer errors

fix(pd): retain unresolved NIXL handles after transfer errors

When a handle reports ERR, the worker performs one complete poll sweep,
retains errored and otherwise unresolved handles, does not retain already-DONE
handles, marks the room Failed, and returns to the worker loop without waiting
indefinitely for a sibling that may remain PROC.

The failed group's outstanding count remains positive, so it cannot become
eligible for deferred ABORT_ACK.

The existing all-DONE path is unchanged.

Mooncake

I also checked the analogous Failed-room skip in Mooncake.

The NIXL-specific failure sequence fixed here does not apply to Mooncake's
normal path:

• chunks for the same destination sessions are assigned to the same transfer
  queue and processed serially;
• the normal transfer path uses synchronous batch_transfer_sync;
• ordinary transfer failures still reach the per-chunk outstanding decrement;
• unexpected worker exceptions terminate the worker instead of surviving to
  process a later same-room Failed skip.

Mooncake is therefore not modified by this PR.

Why unresolved NIXL handles are retained

This PR deliberately does not call release_xfer_handle() on errored or
unresolved handles.

The current NIXL UCX release path does not provide SGLang with a reliable
"abort incomplete" result that could be used as a quiescence proof. Retaining
unresolved ownership is therefore the conservative behavior here.

Transport-backed reclamation for permanently unresolved handles is left as
follow-up work.

Tests

All added regressions are deterministic CPU tests; no GPU, RDMA device, or
installed NIXL runtime is required.

Negative control on upstream main:

FAIL as expected
expected unresolved count: 1
observed count: 0
false ABORT_ACK emitted: yes

Patched branch:

targeted false-ACK regression: 1 passed
ERR/accounting regression file: 7 passed
relevant NIXL/disaggregation suite:
  84 passed, 0 failed, 0 skipped, 2 subtests passed

Also verified:

py_compile: PASS
Black 26.1.0: PASS
isort 7.0.0: PASS
git diff --check: PASS

The new regression file is registered with:

register_cpu_ci(est_time=2, suite="base-a-test-cpu")

No runner_configs.yml change is required.

Not covered / follow-ups

• No reclamation policy is added for permanently unresolved errored handles.
• No observability metric is added for retained unresolved groups.
• The existing deferred Decode-side release timeout is unchanged.
• This PR does not modify NIXL abort/release semantics.
• Open PR [PD] Align defensive protocol behavior across Mooncake, NIXL, and Mori #35281 introduces additional stale-room cleanup paths. I separately
  validated that those paths must preserve the same per-chunk accounting
  invariant; the [PD] Align defensive protocol behavior across Mooncake, NIXL, and Mori #35281-specific compatibility change is intentionally not
  included here because it is not on main.

Checklist

• Format with Black/isort
• Add deterministic unit tests
• Existing relevant disaggregation tests pass
• Accuracy benchmark — N/A: abort-path correctness; no model-math change
• Speed benchmark — N/A: no inference hot-path change
• Code style

Related: #35360, #35049, #35281, 30a50a2

---

CI States

Latest PR Test (Base): ❌ Run #33084550496
Latest PR Test (Extra): ❌ Run #33084550468
Latest PR Test (AMD ROCm 7.2): ❌ Run #33084552980

[200lz]

Hi @ShangmingCai @YAMY1234 — this restores the deferred-release accounting invariant from #35360 / 30a50a2. I included a current-main negative control: an unresolved count of 1 is erased by a later Failed-room skip, producing a false ABORT_ACK; the patched path leaves the count at 1 and does not ACK. The PR is split so the per-chunk accounting fix is independently reviewable from the NIXL ERR-handle retention change. Thanks for taking a look.