[NickLucche]

Purpose

HiSparse currently rejects host imports whose computed-token count is not block aligned. This draft explores an alternative to #56946: receive the full final block and restore its contents into one writable GPU page, so a successful NIXL full-prompt import resumes with a single-token forward instead of prefilling the unaligned tail on D.

The idea is that we want to avoid having to run prefill on D, however short they might be (<block_size here), with north star being deploying D with decode-only CG (aligned with rest of cases).

NIXL analogy: transfer capacity is not sequence length

NIXL already transfers physical blocks containing partially filled prompt tails. Moving the unused bytes does not make them computed tokens. HiSparse can use the same distinction:

For a 32,019-token prompt, block size 128	This approach
Real imported tokens	32,019
Host blocks received	251
Physical capacity	32,128 token slots
Unused tail capacity ("holes")	109 token slots
Prompt work on D after a full import	The usual last-token replay for logits

The holes are unused rows within the request's final host block. They do not change token positions or sequence length, and generation fills them at their normal offsets. P does not compute dummy padding tokens. No sub-block allocator or NIXL wire-protocol change is required.

Implementation and tradeoff

• Reserve one writable resident page for the final imported block, including aligned prompts whose last token is replayed.
• After a successful external receive, enqueue a full-page host-to-device restore on every TP rank. Copies run on the compute stream before the forward/CUDA-graph replay.
• Retain both endpoints until every participating worker completes its copy. Avoid publishing an uninitialized resident copy to prefix-cache users, and keep an incomplete host page unsealed.
• Register a completed, valid restored GPU page for later prefix hits only after all workers finish. Pending restores remain hidden, and incomplete pages and aborted requests are excluded. This incorporates NickLucche/vllm#12.
• Continue using the existing row-mirror and page-sealing paths for newly generated KV.

Compared with #56946, this transfers the final block and pays for one page of H2D copying per layer/rank rather than recomputing up to block_size - 1 prompt tokens. It also reserves a resident tail page for aligned imports. The unused host rows are ordinary tail capacity, not extra logical tokens or permanent gaps between prompt and output. No performance improvement has been measured.

The intended benefit is to keep successful full-prompt P/D handoffs on single-token execution, compatible with decode-only graph capture. This is not a guarantee that D can never prefill: cache misses, failed transfers, preemption, or partial external hits can still require recomputation.

Related work

Checked #56699 and its comments, open PRs mentioning it, and open PRs matching HiSparse import, landing, tail, partial, full block, and tail restore.

• [Bugfix][HiSparse][NIXL] Recompute unaligned host-import tails locally #56946 addresses the same admission failure by rounding the imported prefix down and prefilling the remainder locally. This draft is the explicitly requested alternative that retains all real prompt KV and changes tail residency instead.
• The notify-only read and empty-region shape fixes are reused verbatim from [Bugfix][HiSparse][NIXL] Recompute unaligned host-import tails locally #56946 (originally [6/N] HiSparse: support direct GPU landing for P/D transfers #55398). They prevent empty local destinations from entering region expansion when P and D have different cache-group layouts. They are independent of tail policy; empty receives notify P, and report finished_recving only when awaiting_kvs=True.
• [6/N] HiSparse: support direct GPU landing for P/D transfers #55398 chooses direct GPU landing for imported history when capacity permits. This draft preserves host landing and restores only the final page.
• [Draft][HiSparse] Evictable resident pages; host pool as an ordinary block pool (on top of #53781) #56109 concerns broader pool/residency management. This draft uses the current pool model.

Related to the deployment discussion in #56699; this does not claim to fix that issue's CUDA launch failure or establish end-to-end P/D correctness.

Test Plan

Exercise aligned, unaligned, and sub-block imports; last-token resumption; partial-page publication; multi-worker completion and abort lifetime; and tail-copy contents on both host-writer and non-writer ranks.

Before marking ready, run real NIXL P/D on NVIDIA GPUs with decode-only graphs, check generated-token/logprob parity against the prefiller reference, and exercise page-boundary crossing, eviction/preemption, and concurrent traffic. Heterogeneous TP/DCP coverage remains pending.

Test Result

.venv/bin/python -m pytest \
  tests/v1/core/test_prefix_caching.py \
  tests/v1/kv_connector/unit/test_hisparse_connector.py \
  tests/v1/worker/test_utils.py \
  -k 'hisparse or connector_completes_partial_prefix' -q

99 passed, 2 skipped, 117 deselected on macOS. The two skipped cases require CUDA. Commit-time pre-commit hooks passed, including the configured mypy hook.

Rerun after integrating #12 and rebasing onto upstream/main at fc8132a5e5: 99 passed, 2 skipped, 117 deselected. The existing import test now checks that one worker completion does not publish the GPU copy and that full completion publishes aligned tails. Pre-commit hooks passed across all PR files. GPU/model validation of the combined change remains pending.

Additional existing empty-receive checks after restoring those two NIXL hunks:

.venv/bin/python -m pytest tests/v1/kv_connector/unit/test_nixl_connector.py \
  -k empty_recv_is_reported_only_when_awaited -q

4 passed, 130 deselected on macOS. The separate completion-time positional group-pairing workaround reported by the GPU investigation is not included in this draft.

After rebasing onto upstream/main (c6fa1f05d1, including #57027), fixed the H200 CI failure in test_hisparse_spill_and_prefix_restore[offload]: the sync checker reports the tail copy as potentially synchronous because PyTorch does not recognize the host pool's external cudaHostRegister registration. A narrow gpu_sync_allowed() scope covers that copy; the transfer and stream ordering are unchanged. The existing CUDA tail-restore test now allocates the host pool through the production registration path and enables strict sync checking.

.venv/bin/python -m pytest tests/v1/worker/test_utils.py \
  tests/v1/core/test_prefix_caching.py -k hisparse -q
.venv/bin/python -m pytest tests/v1/kv_connector/unit/test_nixl_connector.py \
  -k empty_recv_is_reported_only_when_awaited -q

Post-rebase results on macOS: 88 passed, 2 CUDA cases skipped and 4 passed, respectively. Pre-commit hooks passed across all PR files. The CUDA regression and failing GPU e2e test still require the new CI run.

Model evaluation / GPU serving: not run. This is a draft for design review, published at the submitter's explicit request without waiting for GPU testing. No accuracy, graph-replay, throughput, or latency claim is made.

AI assistance (OpenAI Codex) was used for implementation, tests, and this description. Human line-by-line review and human-run GPU validation remain pending before this draft is ready for merge.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[NickLucche]

/ci run

[github-actions]

❌ This PR is 4 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

[NickLucche]

/ci run

[github-actions]

✅ Triggered Buildkite CI #89167 for commit 29e3748ffdd8.

[LucasWilkinson]

Makes sense to me thanks for doing this! minor issue Astra found: NickLucche#12

[NickLucche]

/ci run

[LucasWilkinson]

/ci run

[github-actions]

❌ This PR is 12 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

[LucasWilkinson]

/ci run

[NickLucche]

/ci retry

[github-actions]

✅ Queued 5 failed job(s) for retry in Buildkite CI #89355.

[NickLucche]

/ci run

[NickLucche]

/ci retry

[github-actions]

✅ The previous CI build is still running: https://buildkite.com/vllm/ci/builds/89355

[NickLucche]

/ci run

[github-actions]

✅ Triggered Buildkite CI #89443 for commit e0ec0db4943f.

[khluu]

/ci retry

[github-actions]

✅ Queued 1 failed job(s) for retry in Buildkite CI #89443.

[NickLucche]

/ci retry

[github-actions]

✅ Triggered Buildkite CI #89629 for commit 3d83c84480e7, running 1 failed step(s) from Buildkite CI #89443.