Skip to content

feat: AMD GPU umbrella execution provider integration - #2376

Draft
Wang, Zhenze (zz002) wants to merge 2 commits into
microsoft:rel-0.15.1from
zz002:feat/amdgpu-ep-0.15.1
Draft

feat: AMD GPU umbrella execution provider integration#2376
Wang, Zhenze (zz002) wants to merge 2 commits into
microsoft:rel-0.15.1from
zz002:feat/amdgpu-ep-0.15.1

Conversation

@zz002

Copy link
Copy Markdown

Summary

Integrates the AMD GPU umbrella execution provider (amdgpu-ep, a plugin EP shipped by the ROCm hip-ep project) into GenAI on top of rel-0.15.1, adding DeviceType::AMDGPU and a device interface that registers and selects the EP through the plugin EP (V2) API.

What

src/amdgpu/ holds the new device interface. It resolves the EP library from AMDGPU_EP_PATH, then next to onnxruntime-genai.dll, onnxruntime.dll and the executable, before falling back to the working directory, and registers it on the interface's OrtEnv (tolerating an already-registered library from an earlier init cycle). SetupProvider picks the EP devices reported as GPUs and appends the EP with the model's provider options; the same options are mirrored to ep.amdgpu.* session config entries, which is how the umbrella EP selects its backend. ShapeInitSessionProviderOptions forwards those options to the device-init session so the allocator can be created there, and GetMemoryInfo() builds the memory info with CreateMemoryInfo_V2 using the AMD vendor id and GPU device type the EP factory registers with.

The umbrella EP's allocator is host- and device-accessible on the AMD APU iGPU, so model inputs stay in device memory and the device is on the continuous-decoding list.

Status

This is a draft opened as a patch source for AMD's downstream CI, which currently builds GenAI from rel-0.15.1 plus this branch. It overlaps in intent with #2165 (AMD GPU EP support); once that lands and covers the umbrella EP's provider-option plumbing, this PR will be closed and the downstream CI switched over. It supersedes #2194, which targeted v0.14.0.

Test plan

  • Windows build against ORT 1.27 (--no_telemetry, Ninja, Release)
  • model_benchmark on Llama-3.1-8B (AWQ int4) with "AMDGPU": {"profile": "hip"} and session.disable_cpu_ep_fallback=1: matches the v0.14.0 + Add AMDGPU execution provider support #2194 baseline on time-to-first-token and token generation throughput

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

ROCm CI and others added 2 commits August 5, 2026 04:37
Adds an AMDGPU device interface that registers the AMD GPU umbrella execution provider
library (amdgpu-ep) on the OrtEnv and selects it through the plugin EP (V2) API.

The interface resolves the EP library from AMDGPU_EP_PATH, then next to
onnxruntime-genai.dll, onnxruntime.dll and the executable, before falling back to the
working directory. Provider options are passed both as plugin EP options and mirrored to
"ep.amdgpu.*" session config entries, which is how the umbrella EP selects its backend;
ShapeInitSessionProviderOptions forwards them to the device-init session as well so the
allocator can be created. The allocator memory info is built with CreateMemoryInfo_V2 using
the AMD vendor id and GPU device type the EP factory registers with.

Co-authored-by: Cursor <cursoragent@cursor.com>
The umbrella EP allocator is host- and device-accessible on the AMD APU iGPU, so model
inputs can stay in device memory and the KV cache stays valid across AppendTokens calls.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant