[benenzhu]

Overview

vLLM's ROCm image now ships the HIP 7.16 runtime (docker/Dockerfile.rocm_base, ROCM_RUNTIME_COMMIT 25e14349f2). On it, hipMemcpyBatchAsync runs SimpleCPUOffloadConnector's host <-> device batches as a blit kernel on the CUs (__amd_rocclr_copyBufferBatch), so every load or store slows the decode that runs next to it. With hipMemcpyFlagExtPreferCE (0x100) in the batch attributes, the runtime hands the whole batch to the SDMA engines instead and no kernel runs.

build_params now sets that flag on HIP >= 7.16 (_copy_flags_for_hip_version, next to the existing numAttrs gate). Older runtimes keep flags 0, so nothing changes for them. CUDA is untouched.

Claims

• Decode next to the copies: up to +122% slower (median) without the flag, +0.4% to +3.9% with it (table below).
• Copy bandwidth is unchanged: 44-56 GB/s in both directions, alone and under load.
• Round trips stay byte-exact.

Validation

MI355X, one GPU, image vllm/vllm-openai-rocm:nightly-40941f2c4ba96fba0a7e282a5355e89cd74d9ac5 (main of 2026-11-03, HIP 7.16.26361). Before = the image's own vLLM (flags 0x0); after = this PR's cuda_mem_ops.py copied over it (flags 0x100); same job, same GPU. The copies go through build_params + copy_blocks exactly as the connector calls them, on one TP4 rank's KV layout of a 60-layer model: 131 per-layer tensors, 16 / 32 KiB pages (3.09 MiB per block), a registered host buffer, shuffled block ids, a low-priority stream. Decode is a CUDA graph of 32 bf16 GEMMs (M x 6144 x 4096 and back, 1.5 GiB of distinct weights, HBM-bound) replayed while N-block transfers loop; slowdown = per-replay time vs the same graph alone.

M	direction	N blocks	decode slowdown before: median (mean, p99)	after: median (mean, p99)
16	load	32	+27.3% (+93.9%, +196%)	+3.6% (+3.3%, +6%)
16	load	128	+3.7% (+46.2%, +455%)	+3.9% (+3.6%, +6%)
16	store	32	+88.2% (+98.3%, +327%)	+0.5% (+0.5%, +2%)
16	store	128	+0.6% (+55.6%, +464%)	+0.6% (+0.6%, +2%)
64	load	32	+121.5% (+75.6%, +132%)	+1.3% (+1.2%, +2%)
64	load	128	+1.8% (+43.4%, +328%)	+1.5% (+1.4%, +2%)
64	store	32	+114.4% (+65.1%, +119%)	+0.4% (+0.4%, +1%)
64	store	128	+0.7% (+43.0%, +338%)	+0.4% (+0.4%, +1%)

Without the flag the slowdown is bimodal: some replays are clean and the rest (10-78% of them, depending on the row) run 1.2x to 5.6x slower, so some medians look fine while the means and tails do not. With the flag no replay is more than 1.2x slower.

• Copy bandwidth, before -> after: load 44.2-55.5 -> 44.1-55.6 GB/s, store 44.7-54.7 -> 44.1-55.1 GB/s (8-512 blocks, alone).
• Byte-exact round trip (64 random blocks x 131 tensors, GPU -> CPU -> other GPU blocks): 0 differing tensors before and after.
• pytest tests/v1/simple_kv_offload/test_hip_mem_ops.py: the new test_copy_flags_for_hip_version covers the gate (unknown, ROCm 7.2, HIP 7.15, 7.16, 8.0); like the rest of that file it needs no HIP runtime.

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

AI assistance (Claude Code) was used to prepare this change.

🤖 Generated with Claude Code