Fix disaggregated prefill cache leak after abort

[kflansburg]

Motivation

An aborted disaggregated prefill request can lose scheduler ownership when an overlapped middle or final chunk result arrives after AbortReq or bootstrap failure. The result path currently drops aborted middle chunks and does not check final chunks before caching and enqueueing them.

For hybrid Mamba models this leaves one request allocation unaccounted for: one full prefill chunk plus the request Mamba state and overlap buffers. The observed production invariant failure was a 16,384-token full-KV deficit and three missing Mamba states.

Modifications

• Treat abort as terminal before optimistic-bootstrap retry.
• Retire aborted final results and abandoned middle results through one idempotent cleanup helper.
• Release metadata, HiCache request state, KV pages, and Mamba state without inserting the aborted request into radix cache.
• Defer cleanup while a later overlapped chunk is still running.
• Ignore delayed results when bootstrap-failure cleanup already released the request.
• Stream the terminal abort exactly once from the result path that owns cleanup.

Accuracy Tests

No model-output behavior changes for successful requests.

• 9 focused CPU tests passed, including delayed middle result, delayed final result, already-cleaned bootstrap failure, and unified-memory move-gate coverage.
• All file-scoped pre-commit hooks passed.

GPU reproduction on the original GLM-5.3-Flash B300 deployment is still pending.

Speed Tests and Profiling

Not applicable. The new checks only run for completed disaggregated-prefill results; no model-forward path changes.

Checklist

• Format code with pre-commit.
• Add unit tests.
• Update documentation (not needed for this lifecycle bug fix).
• Provide accuracy and speed benchmarks (not applicable; production GPU reproduction remains pending).
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): ❌ Run #33272687086
Latest PR Test (Extra): ❌ Run #33272686834
Latest PR Test (AMD ROCm 7.2): ⏳ Run #33272687029

[rwang5203]

Normal abort cleanup now works, but abort() can still raise before metadata, HiCache, KV, and Mamba release. It may be worth making transport notification best effort so every local cleanup step still completes.

[rwang5203]

The cleanup closes the reproduced ownership gap, but it and PR 36724 both modify the disaggregated prefill result loop. Combining them currently conflicts around terminal row collection and streaming. Try using a joint regression during conflict resolution, which can cover embedding failures and ordinary aborts while proving each terminal request streams exactly once.

[mickqian]

After #36958, req.kv is non-optional, so both req.kv is not None checks are now always true. Please rebase and key KV ownership on req.is_holding_kv (plus the separate Mamba/metadata ownership), while keeping transport abort best-effort so it cannot bypass local cleanup.

[kflansburg]

Rebased onto current main and addressed the #36958 follow-ups: KV ownership now uses req.is_holding_kv, and sender abort is best-effort so local metadata, HiCache, KV, and Mamba cleanup still completes. The joint exact-once terminal-streaming regression with #36724 is deferred until this lands and #36724 is rebased.

[mickqian]

/tag-and-rerun-ci

[mickqian]

covered in #36988, thanks