Add AMDGPU execution provider support by AMDmoore · Pull Request #2194 · microsoft/onnxruntime-genai

AMDmoore · 2026-06-03T10:15:22Z

Summary

Register a new DeviceType::AMDGPU in OGA so models can target the AMD GPU umbrella execution provider (shipped as amdgpu-ep.dll / libamdgpu-ep.so). The umbrella EP forwards to its backend (currently the HIP/MorphiZen backend), selected at runtime via the "profile" provider option. The integration mirrors the existing RyzenAI path: an AMDGPU namespace + Interface (allocator, dispatch, session-options setup), a dispatch-table entry in session_options.cpp, and a CreateMemoryInfo_V2 code path in EnsureDeviceOrtInit (because the plugin EP's OrtMemoryInfo is registered via the V2 API with AMD GPU vendor id 0x1002, not the legacy OrtMemoryInfo::Create). Both past_present_share_buffer=true and =false work; KV cache routing through the AMD GPU allocator is device-agnostic in DefaultKeyValueCache.
Add DeviceType::AMDGPU to the devices_supporting_continuous_decoding whitelist in Generator::AppendTokens. Without it, multiple AppendTokens calls without an intervening RewindToLength(0) — as used by teacher-forcing PPL evaluation and multi-turn chat — throw "Continuous decoding is not supported on the selected device type". The AMD GPU EP's KV cache lives in UMA-pinned memory (hipHostMallocMapped, host-readable) and supports the same shared-buffer KV cache semantics as CUDA / WebGPU / RyzenAI, so it qualifies for the same whitelist entry.
Allocate AMD GPU model inputs (input_ids, position_ids, attention_mask, etc.) on device memory via the EP's GPU allocator instead of CPU, so ORT can alias the buffers directly (zero H2D/D2H copies for inputs and logits).

Configuration

"provider_options": [{ "AMDGPU": { "profile": "llm" } }]

or

config.append_provider("AMDGPU")

Notes

New source directory src/amdgpu/ ports from src/ryzenai/ with the EP name and vendor filter swapped; behavioural differences are documented inline.
The EP is registered as "amdgpu" and loaded from amdgpu-ep.dll (libamdgpu-ep.so); the path can be overridden with the AMDGPU_EP_PATH environment variable. SetupProvider mirrors provider options into ep.<name>.* session-config entries since the umbrella reads them from there.
EnsureDeviceOrtInit takes the V2 OrtMemoryInfo path for AMDGPU because the legacy OrtMemoryInfo::Create only knows ORT-internal device names ("Cuda", "DML", ...) and would fail with "Specified device is not supported. Try CreateMemoryInfo_V2." for any plugin EP. The device-init dummy session reuses the AMDGPU provider options from genai_config so the umbrella can select its backend; no profile is hardcoded.
The AMD GPU EP itself (the DLL providing the kernels) is shipped separately and is not part of this PR.

Register the MorphiZen execution provider (AMD GPU plugin EP, shipped as onnxruntime_morphizen_ep.dll) so OGA can route models to its allocator and KV cache stays on GPU instead of falling back to host RAM. Mirrors the existing RyzenAI integration: - src/smartptrs.h: add DeviceType::MorphiZenEP. - src/morphizen_ep/{interface,session_options}.{h,cpp} (new): ported from src/ryzenai/. SetupProvider filters OrtHardwareDeviceType_GPU; the EP factory itself restricts to AMD GPU vendor id 0x1002. - src/generators.cpp: to_string()/GetDeviceInterface()/Shutdown() learn the new DeviceType. - src/models/session_options.cpp: dispatch table maps "MorphiZenEP" to MorphiZenEPExecutionProvider::AppendExecutionProvider. - src/models/model.cpp: device_type_names[] / device_memory_type_names[] gain entries; EnsureDeviceOrtInit takes the V2 path for MorphiZenEP (CreateMemoryInfo_V2 with AMD vendor id + GPU type) because OrtMemoryInfo::Create only knows ORT-internal device names. - cmake/global_variables.cmake: add morphizen_ep/* to generator_srcs. Both past_present_share_buffer=true and =false work with this change; KV cache routing through the MorphiZen allocator is device-agnostic in DefaultKeyValueCache.

AMDmoore · 2026-06-03T14:55:22Z

AMDmoore please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]
Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"
Contributor License Agreement

@microsoft-github-policy-service agree company="AMD"

Add DeviceType::MorphiZenEP to the devices_supporting_continuous_decoding whitelist in Generator::AppendTokens. Without this entry, multiple AppendTokens calls without an intervening RewindToLength(0) -- as used by teacher-forcing PPL evaluation and any multi-turn chat workload -- throw 'Continuous decoding is not supported on the selected device type'. MorphiZenEP's KV cache lives in UMA-pinned memory (hipHostMallocMapped) which is host-readable and supports the same shared-buffer KV cache semantics as CUDA / WEBGPU / RyzenAI. The fix mirrors the existing entries for those device types.

Add DeviceType::MorphiZenEP to the input-device whitelist so model inputs (input_ids, position_ids, attention_mask, etc.) are allocated via the EP's GPU allocator instead of CPU. This lets ORT alias the buffers directly (zero H2D/D2H copies for inputs and logits); verified end-to-end on Llama-3.1-8B asym with no decode regression and ~22% faster prefill. Also restore an upstream comment in EnsureDeviceOrtInit that was inadvertently reworded.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds support for a new ONNX Runtime plugin execution provider (“MorphiZenEP”) by introducing a new device type and wiring provider setup / allocator handling into model/session initialization.

Changes:

Added DeviceType entry and mappings to/from provider name strings.
Added MorphiZenEP execution provider integration (session options append + EP interface loader/setup).
Updated allocator initialization logic for MorphiZenEP using ORT’s CreateMemoryInfo_V2, and hooked build system to compile new sources.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/smartptrs.h	Adds a new `DeviceType` enum value for MorphiZenEP.
src/morphizen_ep/session_options.h	Declares MorphiZenEP `AppendExecutionProvider` entry point.
src/morphizen_ep/session_options.cpp	Implements session option wiring for MorphiZenEP.
src/morphizen_ep/interface.h	Declares MorphiZenEP device interface used by the runtime.
src/morphizen_ep/interface.cpp	Implements EP registration/loading and provider setup logic for MorphiZenEP.
src/models/session_options.cpp	Registers MorphiZenEP in provider dispatch table.
src/models/model.cpp	Extends allocator init to support MorphiZenEP allocator via `CreateMemoryInfo_V2`.
src/generators.cpp	Adds MorphiZenEP interface hookup, shutdown, and device type conversions.
cmake/global_variables.cmake	Adds MorphiZenEP sources to the build glob.

AMDmoore · 2026-06-15T09:48:00Z

  OpenVINO,
  NvTensorRtRtx,
  RyzenAI,
+  MorphiZenEP,
  MAX


I checked all the references to DeviceType::MorphiZenEP, they are all correct.

AMDmoore · 2026-06-15T09:57:53Z

+      const OrtEpDevice* const* devices = nullptr;
+      size_t ndevices = 0;
+
+      Ort::ThrowOnError(Ort::api->GetEpDevices(&GetOrtEnv(), &devices, &ndevices));
+
+      for (const auto& device : std::span{devices, ndevices}) {


It's owned by ORT and will not be released by caller.

…n_ep Address PR review: dlopen(RTLD_NOLOAD) still bumps the refcount of an already-loaded library, so dlclose the returned handle before returning. Also add <string>/<utility>/<vector> and smartptrs.h includes to interface.h instead of relying on transitive includes. Co-authored-by: Cursor <cursoragent@cursor.com>

…#12) Rename the OGA execution provider integration from MorphiZenEP to AMDGPU so it targets the AMD GPU umbrella EP (amdgpu-ep.dll) instead of loading onnxruntime_morphizen_ep.dll directly. The umbrella forwards to the HIP backend, selected via the "profile" provider option. - Rename src/morphizen_ep -> src/amdgpu and DeviceType::MorphiZenEP -> DeviceType::AMDGPU across generators.cpp, model.cpp, the session_options dispatch table, smartptrs.h, and the cmake source glob. - interface.cpp: register the EP as "amdgpu", load amdgpu-ep.dll (libamdgpu-ep.so), env override AMDGPU_EP_PATH. SetupProvider also mirrors provider options into ep.<name>.* session config entries since the umbrella reads them from there. - EnsureDeviceOrtInit: the device-init dummy session reuses the AMDGPU provider options from genai_config so the umbrella can select its backend; no profile is hardcoded. genai_config.json selects it via: "provider_options": [{ "AMDGPU": {"profile": "llm"} }] Co-authored-by: Cursor <cursoragent@cursor.com>

pull/<n>.patch is a multi-commit mbox. `git apply` pre-checks every hunk against the base tree at once, so a PR whose own series adds -> modifies -> renames the same path (microsoft/onnxruntime-genai#2194: src/morphizen_ep/* -> src/amdgpu/*) self-conflicts and the whole build fails. Replay each commit in order with `git am` instead (set a committer identity since the runner has none), which applies the series cleanly. Bump the OGA cache key (-am) so the stale git-apply artifact is not reused. The renamed AMDGPU integration dlopen's libamdgpu-ep.so while the project still builds the EP as libhipgpu.so, so copy it under the expected name in the bundle step (dep-staging + closure verification then cover it). Co-authored-by: Cursor <cursoragent@cursor.com>

kunal-vaishnavi · 2026-06-30T07:47:55Z

+      const OrtEpDevice* const* devices = nullptr;
+      size_t ndevices = 0;
+
+      Ort::ThrowOnError(Ort::api->GetEpDevices(&GetOrtEnv(), &devices, &ndevices));


Can we use the helper APIs from onnxruntime_inline.h instead of making direct Ort::api calls?

onnxruntime-genai/src/models/onnxruntime_inline.h

Lines 138 to 140 in 6be5300

inline void GetEpDevices(OrtEnv* env, const OrtEpDevice* const** device_ptrs, size_t* num_devices) {

ThrowOnError(api->GetEpDevices(env, device_ptrs, num_devices));

}

kunal-vaishnavi · 2026-06-30T07:49:02Z

+    }
+
+    for (size_t i = 0; i < config_keys.size(); ++i) {
+      Ort::ThrowOnError(Ort::api->AddSessionConfigEntry(


Same comment as above for the other direct Ort::api calls in this PR.

onnxruntime-genai/src/models/onnxruntime_inline.h

Lines 678 to 680 in 6be5300

inline OrtSessionOptions& OrtSessionOptions::AddConfigEntry(const char* config_key, const char* config_value) {

Ort::ThrowOnError(Ort::api->AddSessionConfigEntry(this, config_key, config_value));

return *this;

kunal-vaishnavi · 2026-06-30T07:52:45Z


  // Names for the device types used by 'SetProviderSessionOptions'
-  static const char* device_type_names[] = {"CPU (Not used, see above)", "cuda", "DML", "WebGPU", "QNN", "OpenVINO (Not used, see above)", "NvTensorRtRtx", "RyzenAI"};
+  static const char* device_type_names[] = {"CPU (Not used, see above)", "cuda", "DML", "WebGPU", "QNN", "OpenVINO (Not used, see above)", "NvTensorRtRtx", "RyzenAI", "AMDGPU"};


Same comment as here regarding simplification. We could create new APIs such as GetProviderName() and GetMemoryInfoName() on DeviceInterface instead of using parallel string arrays.

kunal-vaishnavi · 2026-06-30T07:58:29Z


  // Get the allocator from the OrtSession for the DeviceType (it's called 'AllocatorCreate' but it's really 'AllocatorGet')
  auto name = device_memory_type_names[static_cast<int>(type)];
  try {


I see several conditionals are getting introduced on a per-EP basis for getting an OrtMemoryInfo object. Can we move this per-EP work to the device interface and make it more generic here?

kunal-vaishnavi · 2026-07-22T00:44:26Z

Closing this PR in favor of #2165

AMDmoore force-pushed the dynamic-shape-morphizen-pr branch from 8f3b79d to a196167 Compare June 3, 2026 10:29

AMDmoore added 2 commits June 4, 2026 02:39

AMDmoore force-pushed the dynamic-shape-morphizen-pr branch from 966d176 to ca26c92 Compare June 12, 2026 04:08

AMDmoore changed the title ~~Add MorphiZen execution provider and OrtIoBinding-based State::Run fast path~~ Add MorphiZen execution provider support Jun 12, 2026

AMDmoore force-pushed the dynamic-shape-morphizen-pr branch from 2256664 to 1e85eec Compare June 12, 2026 08:10

AMDmoore marked this pull request as ready for review June 12, 2026 08:13

AMDmoore requested a review from a team as a code owner June 12, 2026 08:13

Copilot AI review requested due to automatic review settings June 12, 2026 08:13

Copilot AI reviewed Jun 12, 2026

View reviewed changes

AMDmoore and others added 2 commits June 15, 2026 04:51

AMDmoore changed the title ~~Add MorphiZen execution provider support~~ Add AMD GPU execution provider support Jun 18, 2026

AMDmoore changed the title ~~Add AMD GPU execution provider support~~ Add AMDGPU execution provider support Jun 18, 2026

Ming Wang (amd-mingw) mentioned this pull request Jun 18, 2026

feat(hipep): add hipep-backend selected via profile=llm onnxruntime/onnxruntime-ep-amdgpu#23

Merged

kunal-vaishnavi reviewed Jun 30, 2026

View reviewed changes

kunal-vaishnavi closed this Jul 22, 2026

Wang, Zhenze (zz002) mentioned this pull request Aug 5, 2026

feat: AMD GPU umbrella execution provider integration #2376

Draft

2 tasks

Navigation Menu

Uh oh!

Add AMDGPU execution provider support - #2194

Add AMDGPU execution provider support#2194
AMDmoore wants to merge 5 commits into
microsoft:mainfrom
AMDmoore:dynamic-shape-morphizen-pr

AMDmoore commented Jun 3, 2026 •

edited

Loading

Uh oh!

AMDmoore commented Jun 3, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

AMDmoore Jun 15, 2026

Uh oh!

Uh oh!

Uh oh!

AMDmoore Jun 15, 2026

Uh oh!

kunal-vaishnavi Jun 30, 2026

Uh oh!

kunal-vaishnavi Jun 30, 2026

Uh oh!

kunal-vaishnavi Jun 30, 2026

Uh oh!

kunal-vaishnavi Jun 30, 2026

Uh oh!

kunal-vaishnavi commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	inline void GetEpDevices(OrtEnv* env, const OrtEpDevice* const** device_ptrs, size_t* num_devices) {
	ThrowOnError(api->GetEpDevices(env, device_ptrs, num_devices));
	}

	inline OrtSessionOptions& OrtSessionOptions::AddConfigEntry(const char* config_key, const char* config_value) {
	Ort::ThrowOnError(Ort::api->AddSessionConfigEntry(this, config_key, config_value));
	return *this;

Uh oh!

Conversation

AMDmoore commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Configuration

Notes

Uh oh!

AMDmoore commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

AMDmoore Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AMDmoore Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AMDmoore commented Jun 3, 2026 •

edited

Loading

AMDmoore commented Jun 3, 2026 •

edited

Loading