perf(dflash): align replicated cache pages with target

[voipmonitor]

Status

Qualified on four RTX PRO 6000 Blackwell GPUs with TP4 and DCP4. The change is stacked on the replicated/sharded cache-group isolation in #519; #519 is stacked on the DFlash DCP implementation in #513.

Resulting behavior

DCP-replicated sliding-window draft cache groups retain the block selected by their attention backend while model KV specs are constructed. During mixed target/draft cache grouping, the planner promotes a smaller draft block to the greatest common physical block size of the sharded target groups only when both invariants hold:

• the backend-selected draft block divides the target alignment, so attention kernels can use the framework block-splitting contract;
• the enlarged natural draft pages fit within the target-determined shared BlockPool stride, so allocated KV bytes do not increase.

Incompatible geometry and candidates that would enlarge the pool preserve the backend-selected draft block. No user-facing configuration flag or model-specific block constant is added. Because the promoted block divides every sharded target block before DCP scaling, scheduler LCM does not increase.

Compatibility impact

The optimization applies only when one cache specification contains both DCP-sharded groups and DCP-replicated sliding-window groups. DCP1, non-windowed replicated attention, caches without mixed replication modes, and memory-costly alignments retain their existing geometry.

Validation

Source qualification used PR commit fd47007a6adc95f1a947d80dc2fd50745e22bdcc in integration commit bfd30c0db01846a1de2e6a47d33aac3fb970b759 (tree 559049e2e214e3091d08138dab7ffe95cb10fd11) over the source-qualified GLM-5.3 stack. B12X was d56c1163b6e019d828ed24f135c2efd05fdca6ea.

Models:

• target: local-inference-lab/GLM-5.3-Flash-NVFP4@520de24eabf507659eaef7c70f14fd584527facc
• draft: local-inference-lab/GLM-5.3-Flash-DFlash2-MXFP8@b6d33aa93fc1ac5b23a88251a1c0ce0bfe2ad17c

Runtime configuration:

physical GPUs: 4,5,6,7
TP=4
DCP=4
CP_KV_CACHE_INTERLEAVE_SIZE=4
SPECULATOR=dflash
NUM_SPECULATIVE_TOKENS=7
DFLASH_ATTENTION_BACKEND=FLASH_ATTN
DFLASH_KV_CACHE_DTYPE=auto
CUDAGRAPH_MODE=FULL
MAX_CUDAGRAPH_CAPTURE_SIZE=128
MAX_NUM_BATCHED_TOKENS=8192
MAX_NUM_SEQS=32
MAX_MODEL_LEN=1048576
DCP_CKV_GATHER=1
B12X_PCIE_ALLREDUCE=1
--kv-cache-memory 12884901888
--additional-config {"glm53_kda_decode_backend":"auto"}

The GLM GDN backend resolves FULL to FULL_DECODE_ONLY; target and DFlash decode graphs were captured through concurrency 16.

At an identical 12 GiB KV budget and 906 physical blocks:

Replicated draft block	Group-aware capacity	Change
1152 control	5,688,681 tokens	—
2304 planner alignment	5,974,904 tokens	+286,223 (+5.03%)

Three 30-second sustained decode runs:

Geometry	CC1 steps/s	CC16 steps/s
1152 control, mean	62.041	330.026
2304 planner alignment, mean	62.055	328.749
Relative change	+0.02%	-0.39%

Raw output throughput is acceptance-dependent. The 2304 runs averaged 139.7/783.0 tok/s at accepted lengths 2.252/2.382 for CC1/CC16; normalized steps/s is the kernel/scheduler comparison signal.

Three 32k standalone prefill captures with 30-second sampling windows measured 10,637, 10,100, and 10,708 prompt tok/s (mean 10,482). The source-qualified 1152 runtime measured 10,529 prompt tok/s under the same 32k methodology.

A 25,210-token prompt followed by 3,072 generated tokens crossed both cache geometries. The warmed 2304 runtime reproduced the 1152 output SHA-256 acf8a33a8123fa87a857b83dcb089deb45f11d69af24010765ed183a92758637. One first request after an empty JIT cache produced a different long-horizon continuation, so the hash is evidence for warmed-path parity rather than a cross-cold-start determinism guarantee.

Tests and static checks:

• 96 passed: tests/v1/core/test_kv_cache_utils.py plus tests/v1/spec_decode/test_dflash_replicated_dcp.py
• ruff check: passed
• ruff format --check: passed

Review requirement

OpenAI Codex assisted with implementation, tests, benchmarking, and pull-request text. Every changed line requires human review before merge.

[coderabbitai]

Warning

Review limit reached

Next included review available in 59 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3ca25d14-a590-4421-9d11-630f168a4323

📥 Commits

Reviewing files that changed from the base of the PR and between 00c4634 and fd47007.

📒 Files selected for processing (2)

• tests/v1/spec_decode/test_dflash_replicated_dcp.py
• vllm/v1/core/kv_cache_utils.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[voipmonitor]

Qualified runtime artifact: voipmonitor/vllm:glm53-flash-nvfp4-dflash2-mxfp8-dcp4-pagealign-vllmbfd30c0d-b12xd56c1163-cu133-torch213-20260829-r2, manifest digest sha256:9a0ce5badb50ac93647bf517573ac5578dee0dc8beccd58e699f44767b14495f, image ID sha256:2897d7a659453f256af37c26fa8aa933885af7cd28c8bda25c85912f8801c94c. The image source-lock label is 7d92e36e7e3d22fa4632f7b91bfdc0612a7ca267c7690d062f1c8cebe0aec26b and resolves integration tree 559049e2e214e3091d08138dab7ffe95cb10fd11. A production-style TP4/DCP4 launch on physical GPUs 4,5,6,7 with automatic 0.95 memory budgeting reported 15.87 GiB available for KV and 7,900,591 group-aware engine KV tokens (7.53 concurrent 1,048,576-token requests). Logs confirmed the 2304-token draft alignment, effective cache-group CP sizes (4,4,4,4,4,1), B12X PCIe all-reduce, B12X target kernels, B12X draft MXFP8 GEMM, FlashAttention 2 draft attention, and target plus DFlash decode graph capture through concurrency 16. A post-start OpenAI chat-completions smoke request completed successfully.

[voipmonitor]

Local pre-commit validation completed for both changed files after the source-qualified image run. All applicable hooks passed, including ruff check, ruff format, typos, Python 3.10 mypy, SPDX headers, lazy-import validation, forbidden-import validation, torch.cuda API validation, configuration-field validation, and the repository suggestion gate. The GitHub pre-run-check remains queued because no runner has accepted the job; it is not reporting a source failure.

[voipmonitor]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.