Skip to content

Add AMDGPU execution provider support - #2093

Closed
Aditya Lohia (aditya-dl) wants to merge 1 commit into
microsoft:mainfrom
aditya-dl:amd/dev/adilohia/amdgpu_support
Closed

Add AMDGPU execution provider support#2093
Aditya Lohia (aditya-dl) wants to merge 1 commit into
microsoft:mainfrom
aditya-dl:amd/dev/adilohia/amdgpu_support

Conversation

@aditya-dl

Copy link
Copy Markdown

Add AMD GPU (MIGraphX) execution provider support to ONNX Runtime GenAI. The provider is exposed as "amdgpu" to users and maps to the MIGraphX EP in ONNX Runtime internally.

Changes:

  • Create src/amdgpu/session_options.{h,cpp} with AppendExecutionProvider that tries V2 plugin path then falls back to V1 legacy API
  • Add provider name normalization ("amdgpu" -> "AMDGPU") and register in the dispatch table
  • Enable graph capture for AMDGPU to allow compiled graph reuse during token generation
  • Add static input shape padding (prompt_gen_ flag) so the EP avoids recompilation on varying prompt lengths. Gated behind NeedsStaticInputShapes() to only activate for AMDGPU, not other EPs
  • Fix input_ids padding to handle both int32 and int64 element types, and correct per-row copy for batch_size > 1
  • Fix position_ids padding to prevent out-of-bounds read on next_tokens when tensor shape is padded to max_length for batch_size > 1

Configuration:
"provider_options": [{ "amdgpu": {} }] or: config.append_provider("amdgpu")

Known limitations:

  • Beam search not supported (requires past_present_share_buffer=true which requires num_beams=1)
  • Inputs allocated on CPU; the MIGraphX EP handles CPU<->GPU transfers internally

@aditya-dl
Aditya Lohia (aditya-dl) force-pushed the amd/dev/adilohia/amdgpu_support branch from c25173d to f185537 Compare April 21, 2026 00:15

#include "../models/session_options.h"

namespace Generators::AMDGPUExecutionProvider {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why name the provider as AMDGPU instead of MIGraphX?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving forward, we're renaming the EP to AMDGPU Execution Provider. Internally, OGA maps "AMDGPU" to "MIGraphX" when communicating with ORT, so there are no ORT-side changes needed. Users configure "amdgpu": {} in their genai_config.json or call config.append_provider("amdgpu").

@baijumeswani Baiju Meswani (baijumeswani) Apr 24, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not rename it inside ORT as well? Having multiple names is a problem because there are multiple components that are execution provider aware (including foundry local and foundry local catalog). It would be problematic if on the surface it appeared that the name is AMDExecutionProvider but when we query OrtGetEpDevices, it returns MIGraphXExecutionProvider?
If the name needs to change, it should change in ort as well.

Comment thread src/models/model.cpp
for (int i = 0; i < inputs_.size(); i++) {
std::string input_name = input_names_[i];

if (input_name == "input_ids") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A boolean called is_prompt_ already exists for detecting the prefill stage when updating the input ids.

// For beam search
if (is_prompt_ && state_.params_->search.num_beams > 1) {
int row_size = static_cast<int>(shape_[1]);
for (int b = 0; b < shape_[0]; b++) {
int in_offset = (b / state_.params_->search.num_beams) * row_size;
int out_offset = b * row_size;
data_span.subspan(out_offset, row_size).CopyFrom(new_tokens.subspan(in_offset, row_size));
}
} else {
data_span.CopyFrom(new_tokens);
}

Can we set the correct input ids when constructed rather than doing it here right before the model runs?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current approach pads in State::Run() because the padding needs to be coordinated across input_ids, position_ids, and logits - all three must be padded to the same max_length for the shapes to be consistent. Moving input_ids padding to DefaultInputIDs::Update() would require scattering the padding logic across three separate classes (input_ids, position_inputs, logits) rather than keeping it centralized.
Also, is_prompt_ is currently used for beam search token duplicating - overloading it for static shape padding could be confusing.
That said, if you feel the architectural separation is important, we can refactor to move each padding into its respective Update() function. Let me know your thoughts.

Comment thread src/models/logits.cpp
shape_[1] = new_kv_length;
if (state_.prompt_gen_)
{
shape_[1] = state_.params_->search.max_length;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typically, we have used enabling or disabling graph capture to decide on static vs dynamic shapes. Can we use that rather than introducing new boolean feature flags such as prompt_gen_?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We considered using use_graph_capture but it's also true for DML and WebGPU. Gating the static input padding behind use_graph_capture would cause DML/WebGPU to also pad inputs to max_length during prompt processing, which they don't need. They handle static shapes through their own mechanisms (DML has ep.dml.enable_graph_capture session config, WebGPU checks enableGraphCapture provider option). The prompt_gen_ flag, gated behind NeedsStaticInputShapes(), ensures the padding only activates for AMDGPU without affecting other EPs.
Let me know if my understanding is right.

keys.emplace_back(option.first.c_str());
values.emplace_back(option.second.c_str());
}
session_options.AppendExecutionProvider("MIGraphX", keys.data(), values.data(), keys.size());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How come the execution provider has a different name when using append execution provider v1 vs v2?

Comment thread src/ort_genai_c.cpp Outdated
Comment on lines +1091 to +1096
// Map AMDGPU to MIGraphX for ORT compatibility
const char* ort_name = registration_name;
if (std::string_view(registration_name) == "AMDGPUExecutionProvider") {
ort_name = "MIGraphXExecutionProvider";
}
Ort::RegisterExecutionProviderLibrary(&(Generators::GetOrtEnv()), ort_name, fs::path(library_path).c_str());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I mentioned above, this has consequences on other layers in the stack (such as foundry local and foundry local catalog). Please unify the naming.

@aditya-dl

Copy link
Copy Markdown
Author

Baiju Meswani (@baijumeswani) Thank you for the feedback on naming consistency. We understand the concern.

A larger AMDGPU execution provider effort is underway. We are open-sourcing the plugin EP that will use the AMDGPU name consistently across the stack. This PR is the initial OGA-side integration to support that direction.

The MIGraphX name will be retained in legacy ORT for backward compatibility. Once the plugin EP is available, the naming will be unified end-to-end.

Add AMD GPU (MIGraphX) execution provider support to ONNX Runtime
GenAI. The provider is exposed as "amdgpu" to users and maps to the
MIGraphX EP in ONNX Runtime internally.

Changes:
- Create src/amdgpu/session_options.{h,cpp} with AppendExecutionProvider
  that tries V2 plugin path then falls back to V1 legacy API
- Add provider name normalization ("amdgpu" -> "AMDGPU") and register
  in the dispatch table
- Enable graph capture for AMDGPU to allow compiled graph reuse
  during token generation
- Add static input shape padding (prompt_gen_ flag) so the EP avoids
  recompilation on varying prompt lengths. Gated behind
  NeedsStaticInputShapes() to only activate for AMDGPU, not other EPs
- Fix input_ids padding to handle both int32 and int64 element types,
  and correct per-row copy for batch_size > 1
- Fix position_ids padding to prevent out-of-bounds read on
  next_tokens when tensor shape is padded to max_length for
  batch_size > 1

Configuration:
  "provider_options": [{ "amdgpu": {} }]
  or: config.append_provider("amdgpu")

Known limitations:
- Beam search not supported (requires past_present_share_buffer=true
  which requires num_beams=1)
- Inputs allocated on CPU; the MIGraphX EP handles CPU<->GPU
  transfers internally
@aditya-dl

Copy link
Copy Markdown
Author

Superseded by #2165. Per AMD direction we have reverted the EP naming from AMDGPU back to MIGraphX in OGA. The new PR carries the same functional changes (EP registration, graph capture enable, static input shape padding, batch_size>1 fixes for input_ids and position_ids) plus a catalog-name alias for Windows ML EP discovery compatibility. Please direct further review there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants