[dpeabody-amd]

ISSUE ID : AICK-1847

Draft, depends on the hipDNN PRs below for its paged attention engines. The plugin is self-contained under projects/hipdnn/tools/hipdnn_vllm/, next to tools/hipdnn_torch/, and touches no other files. Without the engines below its attention backend falls back to vLLM's Triton kernels; with them, decoder attention runs on hipDNN.

• feat(hipdnn): paged K/V SDPA contract and references #14245: paged K/V and attention-sink SDPA contract and references
• feat(hip-kernel-provider): paged/ragged SDPA engine for gfx942/gfx950 #14247: hipkernel:AttentionPagedUnified, gfx942/gfx950 (stacked on feat(hipdnn): paged K/V SDPA contract and references #14245)
• feat(hip-kernel-provider): paged/ragged SDPA in Gfx1151WmmaAttention #14254: paged/ragged SDPA and sinks in hipkernel:Gfx1151WmmaAttention, RDNA (stacked on feat(hipdnn,hip-kernel-provider): ship the RDNA SDPA catalog on the new rocKE kernels; retire wmma_fmha_fwd #13897)
• fix(hip-kernel-provider): ASM SDPA and HIP flash2 decline attention sinks #14246: ASM SDPA / HIP flash2 decline attention sinks (independent)

Summary

hipdnn_vllm is an out-of-tree vLLM plugin that routes vLLM operations through hipDNN on ROCm. There are no vLLM source changes and no model changes. You pip install the wheel, then run vllm serve <model> --attention-backend HIPDNN. It is not registered upstream in vLLM; it targets users who want to try hipDNN under vLLM on RDNA (gfx110X, gfx1151, gfx120X) or CDNA (gfx942, gfx950).

When hipDNN declines a call (no engine for that graph), the call runs on vLLM's own kernel instead, and a per-process coverage report records every route and decline reason. Output therefore stays correct whatever the installed engines cover. As new provider engines land, more calls route to hipDNN without any plugin change.

What changes

• Registration: one vllm.general_plugins entry point. It is idempotent and does no GPU work at registration.
• Attention backend HIPDNN:
  • Subclasses vLLM's Triton attention backend, so it shares its KV-cache layout and KV writes, and can fall back to it per batch.
  • Decode, speculative decode, and mixed prefill/decode batches run as one paged/ragged hipDNN SDPA graph per layer. The graph reads vLLM's block table, sequence lengths and query offsets zero-copy.
  • Plans are sized per batch, including the batches vLLM captures into CUDA graphs. Nothing is built during capture.
  • Encoder-only layers use a dense path. Attention sinks are passed to hipDNN as sink_token.
• Other ops:
  • vLLM IR hipdnn providers for rms_norm and fused_add_rms_norm, selected with --ir-op-priority.
  • CustomOp overrides for MMEncoderAttention (ViT attention), LayerNorm and Conv2d/Conv3d. Conv is opt-in.
  • Opt-in unquantized linear through hipBLASLt. It patches vLLM's unquantized-GEMM dispatch, with a symbol check that fails loudly if the vLLM version doesn't match.
• C++ extension (csrc/), a torch extension over the hipDNN C++ frontend:
  • per-device handle on the current stream;
  • LRU plan cache that also caches declines;
  • plans used under graph capture are pinned;
  • decline taxonomy;
  • GIL released in execute;
  • lock-free stats counters;
  • HIP errors leaked by backend calls are consumed.
• Packaging:
  • One wheel per GPU family, pinned to a vLLM/torch/ROCm set in BUILD_PINS.toml.
  • Until TheRock ships rocKE-enabled providers and a hipDNN with the changes above, the wheel bundles its own hipDNN backend and providers. _C loads them privately by explicit path (frontend runtime-load mode, RTLD_LOCAL), while torch's bundled copy stays loaded and idle.
  • Includes a Dockerfile.
  • python -m hipdnn_vllm doctor checks the install.
• A local vLLM patch (patches/vllm/0001): replaces a get_name() == "TRITON_ATTN" assert for chunk-lookback models with a supports_chunk_lookback() capability check. It is applied by scripts/apply_vllm_patches.sh and the Dockerfile. It's a candidate for upstreaming.
• Startup: the plugin moves comgr's code-object cache off network filesystems, the same idea as upstream comgr's NFS handling. On some cluster nodes this cut first-handle time several-fold. Opt out with HIPDNN_VLLM_COMGR_CACHE=keep.
• Tests (unit, GPU, e2e) and a validation harness (validation/) for Slurm clusters.
  • The harness measures teacher-forced top-1 agreement with plugin-off runs, judged against the run-to-run agreement of two identical plugin-off runs.
  • It reports a model × arch coverage table and relative throughput.

Behaviour and API changes

• New code only. No existing file in the repository changes.
• At runtime nothing changes unless a user installs the wheel and selects --attention-backend HIPDNN or names the hipdnn IR provider. The default ops are attention, encoder attention, rms_norm, fused_add_rms_norm and layer_norm; linear and conv are opt-in.

Known integration work

• Engines: the paged attention engines come from feat(hipdnn): paged K/V SDPA contract and references #14245, feat(hip-kernel-provider): paged/ragged SDPA engine for gfx942/gfx950 #14247 and feat(hip-kernel-provider): paged/ragged SDPA in Gfx1151WmmaAttention #14254. Until they merge, the wheel build is pointed at a hipDNN tree that contains them (HIPDNN_SRC in scripts/build_wheel.sh).
• Bundled backend: removed once vLLM pins a torch whose ROCm ships hipDNN with these changes, and TheRock enables rocKE in hip-kernel-provider.
• Builders: the general SDPA builder (csrc/graphs/sdpa.cpp) and the encoder-attention builder overlap; consolidating them is a follow-up.
• Throughput:
  • On RDNA, decode uses the prefill kernel until split-KV decode can ship through kernel slots (RFC 0021).
  • On CDNA, the paged kernel is slower per decode step than vLLM's Triton kernel.
• Engine coverage gaps: gfx950 encoder attention (no engine for fused-QKV strides), CDNA head dim 96, softcap and ALiBi. These fall back.
• vLLM pin: a ROCm 10 (TheRock pip) nightly, because the vLLM release has no wheel for that ROCm. Bumping it is a deliberate step that reruns the validation.

Validation (Linux, Slurm compute nodes)

All runs below use this PR's exact commit. They install the release wheels built from it (one per GPU family: gfx94X, gfx950, gfx110X-all, gfx120X-all, gfx1151), with no source overlay. Each wheel bundles a hipDNN built from the engine PRs (#14245, #14247, #14254), including their sink support. Each wheel's build record names this commit, and both trees were clean.

Test suites from the installed wheels

arch	doctor	GPU suite	paged attention (incl. sinks, CUDA-graph capture/replay)	spawn
gfx942	pass	519 passed, 42 skipped, 29 xfailed	58 passed	1 passed
gfx950	pass	519 passed, 42 skipped, 29 xfailed	58 passed	1 passed
gfx1100	pass	584 passed, 1 skipped, 5 xfailed	58 passed	1 passed
gfx1201	pass	584 passed, 1 skipped, 5 xfailed	58 passed	1 passed
gfx1151	pass	584 passed, 1 skipped, 5 xfailed	58 passed	1 passed

• 0 failures and 0 errors on any architecture.
• Skips and xfails on CDNA: dense SDPA and encoder-attention shapes that no CDNA engine serves yet (NO_ENGINE). The remaining xfails, on every architecture, are declared declines such as fp32 encoder attention and row-strided norms. Each case checks that the op falls back correctly.
• Conv: the conv tests (opt-in op) are deselected from these runs.
• Unit tests: 196 passed, 9 skipped (the skipped cases need ROCm; they ran on a CPU node).
• Also on this content: clang-tidy 20 reports no findings on csrc/. During development, a KV pool larger than 4 GiB ran with no fallbacks on gfx1100 and gfx1151 (Gemma-3-4B, strict mode).

End-to-end model matrix (vLLM default scheduler settings, FULL_AND_PIECEWISE CUDA graphs)

Models: Qwen3-8B, Phi-3-mini, Gemma-3-4B, gpt-oss-20b, Qwen2.5-VL-3B and bge-m3, on all five architectures.

Not run (too large for the GPU):

• Qwen3-8B on gfx1201 (16 GB GPU).
• gpt-oss-20b on gfx1201 (16 GB) and gfx1151 (31 GB visible); the model needs 40 GB.

Method: each model is run with the plugin off (the vLLM TRITON_ATTN baseline) and with the plugin's default op set. The accuracy measure is teacher-forced top-1 agreement with the plugin-off run. Two references come from the same job:

• a second plugin-off run, which measures run-to-run noise;
• vLLM's own vllm_c norm kernels, a non-hipDNN swap of the same ops.

Routing:

• Decoder attention runs on hipDNN with no fallbacks for Qwen3-8B, Gemma-3-4B and Qwen2.5-VL on every architecture where they ran. This covers prefill, mixed batches and decode, both eager and CUDA-graph captured.
• gpt-oss-20b attention, including sinks, runs entirely on hipDNN on gfx942, gfx950 and gfx1100.
• Phi-3-mini (head dim 96): RDNA serves it. On CDNA it falls back with NO_ENGINE.
• Encoder attention: bge-m3 and the Qwen2.5-VL ViT run on hipDNN on RDNA and fall back with NO_ENGINE on CDNA.
• Norms: a handful of rms_norm calls (4–6 per model) decline with noncollapsible_rows.
• Every fallback is one of the known gaps listed above and is reported with its reason. No run produced a NaN.

Accuracy:

• These rows clear the run-to-run bar:
  • every Qwen3-8B and Gemma-3-4B run that ran;
  • bge-m3 on every architecture (cosine 1.000);
  • Qwen2.5-VL on gfx942 and gfx1151.
• The remaining rows miss the 0.99 threshold by a few tokens out of about 250–2,000 positions. Most disagreements are near ties. Each miss is the same size as the plugin-off run's own spread or the vllm_c swap:

arch	model	plugin default	plugin off, vLLM default backend	vllm_c swap	second plugin-off run
gfx942	gpt-oss-20b	0.988	0.992	0.992	1.000
gfx950	gpt-oss-20b	0.987	0.987	0.990	1.000
gfx1100	gpt-oss-20b	0.989	1.000	0.989	0.990
gfx942	Phi-3-mini (attention on fallback)	0.984	0.990	0.986	0.986
gfx1151	Phi-3-mini	0.983	0.991	0.988	0.993
gfx950	Qwen2.5-VL-3B (4 of 260 tokens)	0.985	0.992	0.992	0.992
gfx1100	Qwen2.5-VL-3B	0.985	0.985	0.989	0.989
gfx1201	Qwen2.5-VL-3B (4 of 241 tokens)	0.983	0.992	1.000	0.992

Phi-3-mini passes on gfx1100 and gfx1201 (0.990). On gfx950 its accuracy is 0.991, with attention on fallback.

Throughput relative to plugin-off vLLM (TRITON_ATTN):

model	gfx942	gfx950	gfx1100	gfx1201	gfx1151
Qwen3-8B	0.79×	0.73×	0.92×	not run	1.02×
Phi-3-mini	0.90×¹	0.86×¹	1.32×	0.93×	1.33×
Gemma-3-4B	0.74×	0.73×	1.32×	0.99×	1.06×
gpt-oss-20b	0.94×	0.90×	0.94×	not run	not run
Qwen2.5-VL-3B	0.58×	0.68×	0.84×	0.92×	0.89×
bge-m3	1.20×¹	1.19×¹	1.08×	1.08×	1.08×

¹ Attention falls back here, so only the norms run on hipDNN.

• The RDNA ratios above 1 for Phi-3 and Gemma-3 come from the TRITON baseline. On gfx1100 and gfx1151, vLLM's default attention backend is faster than TRITON on these models; against it, the plugin is 0.92–1.03×.
• The CDNA gap is in the paged decode kernel (see Known integration work).
• Throughput comes from single runs on GPUs whose host may run other jobs. Treat differences of a few percent as noise.

ASIC coverage

gfx942, gfx950, gfx1100, gfx1151 and gfx1201, which are vLLM's supported ROCm GPUs. Linux only (vLLM has no native Windows support).

[therock-pr-bot]

❌ PR Check — Action Required

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	❌ Fail	Error: Check concluded with failure.
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

⚠️ 1 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

• ❌ pre-commit

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[therock-pr-bot]

Pre-commit check failed

⛔ pre-commit failed

Please run locally:

• python -m pip install pre-commit
• pre-commit install
• pre-commit run --all-files --show-diff-on-failure

This repo uses .pre-commit-config.yaml.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop   #14255   +/-   ##
========================================
  Coverage    75.95%   75.95%           
========================================
  Files         2880     2880           
  Lines       479080   479080           
  Branches     71971    71971           
========================================
+ Hits        363883   363884    +1     
+ Misses       90339    90338    -1     
  Partials     24858    24858           

Flag	Coverage Δ		*Carryforward flag
TensileLite-CPP	48.93% <ø> (ø)		Carriedforward from 11e3828
TensileLite-Unit	83.68% <ø> (ø)		Carriedforward from 11e3828
hipBLAS	90.62% <ø> (ø)		Carriedforward from 11e3828
hipBLASLt	64.43% <ø> (ø)		Carriedforward from 11e3828
hipCUB	82.68% <ø> (ø)		Carriedforward from 11e3828
hipDNN	86.79% <ø> (+<0.01%)	⬆️
hipFFT	42.00% <ø> (ø)		Carriedforward from 11e3828
hipRAND	76.12% <ø> (ø)		Carriedforward from 11e3828
hipSOLVER	69.67% <ø> (ø)		Carriedforward from 11e3828
hipSPARSE	86.99% <ø> (ø)		Carriedforward from 11e3828
rocBLAS	68.73% <ø> (ø)		Carriedforward from 11e3828
rocFFT	48.10% <ø> (ø)		Carriedforward from 11e3828
rocRAND	57.30% <ø> (ø)		Carriedforward from 11e3828
rocSOLVER	76.74% <ø> (ø)		Carriedforward from 11e3828
rocSPARSE	76.92% <ø> (ø)		Carriedforward from 11e3828
rocThrust	93.15% <ø> (ø)		Carriedforward from 11e3828

*This pull request uses carry forward flags. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.