[Model] Support disk-backed (mmap) PLE table for Qwen3.8-Flash-Next (VLLM_PLE_MMAP)

[Trosfy]

What this changes

Qwen3.8-Flash-Next has a 47.68 GiB FP8 PLE table. On machines with limited host memory, or one shared CPU and GPU memory pool, keeping that table resident can prevent the model from fitting.

VLLM_PLE_MMAP=1 reads the required rows from the checkpoint's safetensors files through read-only mmap. Discovery follows model.safetensors.index.json when present and rejects missing, duplicate, malformed, or unsupported shards and scales.

Model Runner V2 gathers PLE rows during input preparation and copies them into stable GPU buffers before the model runs. Captured code only reads those buffers, so FULL cudagraph modes work without the previous whole-forward splitting op.

The mapped files use Linux's normal file cache. Recently accessed pages may remain resident, but they are clean and file backed, so the kernel can reclaim them when memory is needed. Prewarm and readahead affect which pages enter the cache; they do not pin the complete table in RAM.

Configuration

Variable	Default	Purpose
VLLM_PLE_MMAP	0	Enable mmap-backed PLE
VLLM_PLE_MMAP_WORKERS	32	Gather worker count
VLLM_PLE_MMAP_CHUNK	2048	Rows per gather task
VLLM_PLE_MMAP_PREWARM	0	Warm a memory-bounded portion of the table
VLLM_PLE_MMAP_READAHEAD	0	Limit coalesced ranges sent to posix_fadvise; 0 disables it
VLLM_PLE_MMAP_PINNED	0	Use pinned host memory for CUDA H2D copies
VLLM_PLE_MMAP_SERIAL	0	Run gathers inline at or below this distinct-row count

The rate-limited timing line is PLE mmap input prep. Its ids_d2h_wait_ms field measures host time blocked at the IDs D2H boundary. This includes dependency-ordered accelerator work queued before the copy, so it is not pure transfer or exclusive PLE cost.

Validation

Current branch:

• pytest tests/models/qwen4_exp/test_ple_mmap.py -q: 204 passed
• pytest tests/v1/worker/ -q: 468 passed, 1 skipped
• CUDA tests cover FP8 and BF16 FULL capture, repeated replay with new rows, padded rows, and dummy input preparation.
• Pre-commit hooks on changed files: passed
• pre-commit run mypy-3.12 --all-files --hook-stage manual: passed

Model evaluation:

• On the 8x3090 tester's production port, a 50-item deterministic accuracy battery was byte-identical to the resident baseline: 27/50 with the same margins. This used the previous PIECEWISE split path. Results
• The current input-preparation path has CUDA replay tests that compare FP8 and BF16 outputs across repeated staged batches.

External performance reports also used the previous PIECEWISE split path:

• sm120, TP1: SERIAL reduced gather p50 from 0.73 to 0.22 ms and ITL from 11.92 to 11.45 ms. PINNED was neutral. Results
• sm86, TP2/PP4 downstream port: resident FULL, resident PIECEWISE, and mmap PIECEWISE measured 15, 19, and 28 ms ITL. PP4 remains unsupported upstream. Results
• Intel Arc Pro B70, TP1: moving the synchronization reduced reported PLE time from 93.21 to 0.80 ms without changing ITL. The original field was observing queued expert-offload work. Results

An earlier GB10 run measured 39.7% of the PLE shard pages resident in file cache and no PLE table bytes resident on the GPU.

Scope

This does not duplicate #53899. That implementation keeps the complete PLE table in host memory and uses an offload worker. This PR keeps the checkpoint files as backing storage and has no PLE worker.

Model Runner V2 is required. Pipeline parallelism greater than one is not supported. The implementation assumes Linux file-cache semantics.

AI assistance was used. I reviewed every changed line, verified the reported tests and external evaluation, and take responsibility for the contribution.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[TSUMUGI-XE]

Third topology, and a measurement that I think changes what PLE mmap forward p50 means.

Box: 2× Intel Arc Pro B70 (32 GB), both behind one PCIe switch sharing a single Gen4 x16
upstream, 128 GiB host RAM, TP=1, --cpu-offload-gb 47 --cpu-offload-params experts,
cudagraph_mode=PIECEWISE, this PR at 75f067130. Idle box, VM off, warmup discarded,
median of 10.

1. The p50 is not PLE's cost. forward() times a blocking D2H of the row ids. Putting an
explicit torch.xpu.synchronize() immediately before that timed region:

                     ITL p50     PLE fwd p50   sync_ms   explicit sync
default              111.39 ms      93.21        92.43        --
+ explicit sync      111.52 ms       0.80         0.17       92.01

The D2H itself is 0.17 ms. ITL does not change — the wait moved earlier, it did not go
away. The 92 ms is the device queue draining, and PLE is simply the first thing each step
that demands a synchronization, so it gets billed for it.

2. The drain tracks host streaming, not PLE. Dropping --cpu-offload-params experts
(so the offloader evicts in declaration order):

                     ITL p50     explicit sync   PLE fwd p50
experts              111.52 ms      92.01           0.80
declaration order    328.65 ms     308.53           0.86
delta                 +217.1        +216.5          +0.06

ITL and the drain move together, 1:1. PLE's real work does not move.

3. Demand paging is leaving 2.6× on the link. Same box, same card, 512 MiB H2D:

pinned bulk copy      26.60 GB/s   (link theoretical 31.5)
pageable bulk copy    17.08 GB/s
UVA fault-driven      10.10 GB/s   (0.93 GiB per token / 92 ms)

What I am unsure about is the fix. Batched prefetch after routing would start earlier
and transfer as one large copy, but on this stack UVA cannot place rows in pinned host USM,
so it means replacing demand paging rather than tuning it — and I do not know whether that
belongs here or somewhere else entirely.

Nothing above looks XPU-specific to me: the misattribution comes from where the sync happens,
and demand-paging efficiency is not a platform property. If it is useful, I am happy to run
whatever direction you take on this box — it seems to be the one where the effect is largest.

[Trosfy]

@tomylin890 Thanks for running both follow-ups. The matched resident PIECEWISE result changes the attribution in the useful direction.

Resident FULL at 15 ms versus resident PIECEWISE at 19 ms supports about 4 ms from graph mode. Mmap PIECEWISE at 28 ms leaves another 9 ms associated with the mmap path, but we cannot call all of that graph re-entry cost from this comparison alone.

We have now reconciled the branch with merged #53896 and moved PLE hashing and mmap gathering into input preparation. Rows are copied into a stable GPU buffer before the model runs, the old whole-forward split op is gone, and FULL graph modes are supported under Model Runner V2.

Upstream still supports PP1 only, so I am not asking you to port or rerun this on the TP2/PP4 stack yet. A future comparison needs to start from the reconciled implementation rather than the old split path.

Your disk-KV follow-up also narrows that result. Two roughly 41 GB restores competing for a 32 GiB pool explain the concurrency pressure, while healthy single restores and intermittent dual behavior do not establish mmap as the cause. We will keep that observation scoped accordingly.

Thanks for correcting the earlier attribution and closing both requested arms.

[Trosfy]

@TSUMUGI-XE Thanks for running this. The explicit sync arm caught a mistake in how we were reading our timing line.

We have renamed it to PLE mmap input prep and changed sync_ms to ids_d2h_wait_ms. The description now says that this is the host wait at the IDs D2H boundary, including earlier queued accelerator work. It is not pure copy time or exclusive PLE cost.

On your box, moving the 92 ms wait earlier left ITL unchanged and exposed the PLE path itself at under a millisecond. The experts versus declaration-order result puts the larger delay in generic parameter offload.

We are keeping #54129 scoped to PLE mmap rather than adding XPU and MoE offload work, so I do not think you need to run anything else for this PR. Thanks for adding the control. Without it, we would have kept charging someone else's queue to PLE.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Trosfy.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Trosfy.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork