Zero-copy I/O for plugin EPs with HOST_ACCESSIBLE memory by ericcraw · Pull Request #28037 · microsoft/onnxruntime

ericcraw · 2026-04-10T18:52:47Z

Description

Adds DevicesAreMemoryCompatible() to skip data copies between devices that share memory (CPU <-> HOST_ACCESSIBLE, or HOST_ACCESSIBLE <-> DEFAULT on the same physical device). Applied in feed/fetch copy planning and in BatchOrCopyMLValue.

Overrides GetOrtDeviceByMemType() in PluginExecutionProvider so the allocation planner routes CPU-type I/O through the HOST_ACCESSIBLE allocator when the plugin EP has registered one. This enables the planner to place intermediate tensors (CPU EP -> plugin EP boundary) in HOST_ACCESSIBLE memory, avoiding copies at the partition boundary.

Updates the in-place optimization check in the allocation planner to use UsesCpuMemory() so it recognizes HOST_ACCESSIBLE outputs as CPU-memory-compatible.

Motivation and Context

Remove unnecessary copies for non-cpu HOST_ACCESSIBLE device allocations.

Adds DevicesAreMemoryCompatible() to skip data copies between devices that share memory (CPU <-> HOST_ACCESSIBLE, or HOST_ACCESSIBLE <-> DEFAULT on the same physical device). Applied in feed/fetch copy planning and in BatchOrCopyMLValue. Overrides GetOrtDeviceByMemType() in PluginExecutionProvider so the allocation planner routes CPU-type I/O through the HOST_ACCESSIBLE allocator when the plugin EP has registered one. This enables the planner to place intermediate tensors (CPU EP -> plugin EP boundary) in HOST_ACCESSIBLE memory, avoiding copies at the partition boundary. Updates the in-place optimization check in the allocation planner to use UsesCpuMemory() so it recognises HOST_ACCESSIBLE outputs as CPU-memory-compatible.

yuslepukhin · 2026-04-16T01:56:17Z

There are no unit tests for DevicesAreMemoryCompatible. Given the function has five distinct logical branches (both CPU, CPU + HOST_ACCESSIBLE same device, CPU + HOST_ACCESSIBLE different device, HOST_ACCESSIBLE ↔ DEFAULT same device, incompatible), it should have dedicated unit tests covering each case.

yuslepukhin

🕐

Copilot

Pull request overview

Adds HOST_ACCESSIBLE-aware device selection and copy-planning logic to reduce (or eliminate) unnecessary feed/fetch and boundary copies for plugin execution providers that register host-accessible memory.

Changes:

Override PluginExecutionProvider::GetOrtDeviceByMemType to route CPU I/O mem types through a registered HOST_ACCESSIBLE allocator/device.
Introduce DevicesAreMemoryCompatible() and apply it to feed/fetch copy planning and BatchOrCopyMLValue to skip transfers when devices are deemed memory-compatible.
Update allocation planner CPU-memory checks to use OrtDevice::UsesCpuMemory() (so HOST_ACCESSIBLE is treated as CPU-memory-compatible).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.h	Declares `PluginExecutionProvider` override for mem-type → device mapping.
onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc	Implements HOST_ACCESSIBLE routing for CPUInput/CPUOutput mem types.
onnxruntime/core/framework/utils.cc	Adds memory-compatibility logic and uses it to skip copies + reuse fetch buffers.
onnxruntime/core/framework/allocation_planner.cc	Treats HOST_ACCESSIBLE as CPU-memory-compatible in in-place planning checks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ericcraw · 2026-04-17T00:54:43Z

Thanks for the feedback! I've ran out of time today unfortunately and I'm going to be out until Wednesday next week. Hopefully you'll hear from me again by then. 😄

Co-authored-by: Copilot <copilot@github.com>

…ocator

Co-authored-by: Copilot <copilot@github.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ocator

yuslepukhin

LGTM

ericcraw · 2026-06-02T18:46:20Z

@yuslepukhin, just minor fix on the ort/enforce. Thanks for rerunning the checks, it looks like it's passing now.

nieubank

Thanks for putting this together, I just confirmed that this fixes a long standing issue I've been hitting recently with a certain EP and IoBinding.

### Description Adds memory_info= parameter to OrtValue.ortvalue_from_shape_and_type(), backed by two new C-level factory methods that look up the registered shared allocator via the full OrtMemoryInfo (including mem_type). This is required because the current shared allocator query doesn't include the memory type making HOST_ACCESSIBLE invisible to python. UsesCpuMemory() is used in GetPyObjFromTensor so that tensors in HOST_ACCESSIBLE memory are returned as zero-copy numpy views. ### Motivation and Context Enable zero copy interop between numpy and ortvalue. This is a follow up for #28037

yuslepukhin reviewed Apr 16, 2026

View reviewed changes

Comment thread onnxruntime/core/framework/utils.cc Outdated

yuslepukhin reviewed Apr 16, 2026

View reviewed changes

Comment thread onnxruntime/core/framework/utils.cc Outdated

yuslepukhin reviewed Apr 16, 2026

View reviewed changes

Comment thread onnxruntime/core/framework/utils.cc

yuslepukhin reviewed Apr 16, 2026

View reviewed changes

Comment thread onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc Outdated

yuslepukhin requested changes Apr 16, 2026

View reviewed changes

yuslepukhin requested a review from Copilot April 16, 2026 18:23

Copilot started reviewing on behalf of yuslepukhin April 16, 2026 18:25 View session

Copilot AI reviewed Apr 16, 2026

View reviewed changes

Comment thread onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc Outdated

Comment thread onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc Outdated

Comment thread onnxruntime/core/framework/utils.cc Outdated

yuslepukhin reviewed Apr 16, 2026

View reviewed changes

Comment thread onnxruntime/core/framework/allocation_planner.cc Outdated

yuslepukhin reviewed Apr 16, 2026

View reviewed changes

Comment thread onnxruntime/core/framework/allocation_planner.cc

yuslepukhin mentioned this pull request Apr 16, 2026

Python API for HOST_ACCESSIBLE OrtValue allocation #28038

Merged

First pass addressing review comments

204dcd7

ericcraw force-pushed the host-accessible-allocator branch from cf5a86b to 204dcd7 Compare April 17, 2026 00:53

ericcraw and others added 5 commits April 23, 2026 13:51

Simplify CanSourceSatisfyTarget and add test cases

40b78f5

Add OrtEp::GetMemoryInfoByMemType for plugin EP memory placement

5a83360

Co-authored-by: Copilot <copilot@github.com>

Merge remote-tracking branch 'upstream/main' into host-accessible-all…

34421c2

…ocator

Fix existing tests

e19213b

Co-authored-by: Copilot <copilot@github.com>

Add additional tests for GetMemoryInfoByMemType

24256a4

ericcraw marked this pull request as ready for review April 30, 2026 22:56

ericcraw requested a review from Copilot April 30, 2026 22:57

Copilot AI reviewed Apr 30, 2026

View reviewed changes

Comment thread onnxruntime/core/framework/utils.cc Outdated

Comment thread onnxruntime/core/framework/utils.cc Outdated

Comment thread onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc

Comment thread onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc Outdated

Copilot started reviewing on behalf of ericcraw April 30, 2026 23:11 View session

skottmckay reviewed May 5, 2026

View reviewed changes

Comment thread onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc Outdated

Comment thread include/onnxruntime/core/session/onnxruntime_ep_c_api.h Outdated

skottmckay reviewed May 5, 2026

View reviewed changes

Comment thread include/onnxruntime/core/session/onnxruntime_ep_c_api.h

skottmckay reviewed May 5, 2026

View reviewed changes

Comment thread onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.h Outdated

ericcraw added 2 commits May 5, 2026 18:33

Merge remote-tracking branch 'upstream/main' into host-accessible-all…

627377e

…ocator

Bump since api version to 1.27 for GetMemoryInfoByMemType

dc9fa91

Address review comments

7219dbe

yuslepukhin previously approved these changes Jun 1, 2026

View reviewed changes

Slightly relax assert to account for control flow cases

5313639

ericcraw dismissed yuslepukhin’s stale review via 5313639 June 1, 2026 21:26

ericcraw requested a review from yuslepukhin June 2, 2026 18:43

nieubank approved these changes Jun 2, 2026

View reviewed changes

yuslepukhin approved these changes Jun 2, 2026

View reviewed changes

yuslepukhin merged commit a827f9c into microsoft:main Jun 2, 2026
86 checks passed

BrewTestBot mentioned this pull request Jun 19, 2026

onnxruntime 1.27.0 Homebrew/homebrew-core#288892

Merged

Uh oh!

Conversation

ericcraw commented Apr 10, 2026

Description

Motivation and Context

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yuslepukhin commented Apr 16, 2026

Uh oh!

yuslepukhin left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ericcraw commented Apr 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yuslepukhin left a comment

Choose a reason for hiding this comment

Uh oh!

ericcraw commented Jun 2, 2026

Uh oh!

nieubank left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants