Add update_inplace overload accepting OrtValue for device-to-device copy - #28256
Merged
Tianlei Wu (tianleiwu) merged 7 commits intoApr 29, 2026
Merged
Conversation
Add a second overload of update_inplace that accepts an OrtValue as source, enabling device-to-device memory copy (e.g., GPU to GPU) without requiring data to go through CPU. This is useful for CUDA graph scenarios where inputs need to be updated in fixed memory addresses. Changes: - onnxruntime_pybind_ortvalue.cc: Add pybind overload for OrtValue source - onnxruntime_inference_collection.py: Update Python wrapper to accept both numpy arrays and OrtValues - onnxruntime_test_python_cudagraph.py: Add tests for OrtValue-to-OrtValue update_inplace across different device combinations Agent-Logs-Url: https://github.com/microsoft/onnxruntime/sessions/a05648d1-3616-4d6e-a448-b8d506c55854 Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Add API to update OrtValue inplace from CUDA buffer
Add update_inplace overload accepting OrtValue for device-to-device copy
Apr 28, 2026
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
added 2 commits
April 28, 2026 11:19
- Create UpdateOrtValueInplace() helper in onnxruntime_pybind_mlvalue for code reuse - Simplify Python binding to call helper instead of duplicating logic - Supports device-to-device transfers (CPU, GPU, DML, CANN, MIGraphX) - Addresses review feedback on PR #28256
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an OrtValue-to-OrtValue overload for OrtValue.update_inplace to enable direct device-to-device updates (notably GPU↔GPU) without requiring a NumPy (CPU) intermediate, and extends Python tests to cover the new copy paths.
Changes:
- Add a new pybind
update_inplace(OrtValue)overload and implementpython::UpdateOrtValueInplace(dst, src)device-aware copy logic. - Update the Python
OrtValue.update_inplacewrapper to accept either a NumPy array or anOrtValue. - Add Python tests for CPU↔CPU, GPU↔GPU, CPU→GPU, and GPU→CPU
OrtValuecopy scenarios (CUDA).
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| onnxruntime/test/python/onnxruntime_test_python_cudagraph.py | Adds tests validating update_inplace with OrtValue sources across CPU/GPU (CUDA) copy directions. |
| onnxruntime/python/onnxruntime_pybind_ortvalue.cc | Exposes a new update_inplace overload accepting an OrtValue source. |
| onnxruntime/python/onnxruntime_pybind_mlvalue.h | Declares UpdateOrtValueInplace(dst, src) helper for in-place OrtValue tensor updates. |
| onnxruntime/python/onnxruntime_pybind_mlvalue.cc | Implements UpdateOrtValueInplace with plugin EP copy support + EP-specific fallbacks (CUDA/MIGraphX/DML/CANN). |
| onnxruntime/python/onnxruntime_inference_collection.py | Updates Python wrapper dispatch to route OrtValue sources to the new C++ overload. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
lintrunner found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
Tianlei Wu (tianleiwu)
force-pushed
the
copilot/update-ortvalue-inplace-cuda-graph
branch
from
April 29, 2026 00:58
a7148a3 to
e1f90a6
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The is_dml_device lambda only checked OrtDevice::GPU with Microsoft vendor, but DML tensors can also use the dedicated OrtDevice::DML device type. This caused CPU↔DML copies through the OrtValue overload to incorrectly fall through and throw 'Unable to copy...'.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Tianlei Wu (tianleiwu)
approved these changes
Apr 29, 2026
Tianlei Wu (tianleiwu)
requested a review
from Hariharan Seshadri (hariharans29)
April 29, 2026 16:53
Hariharan Seshadri (hariharans29)
approved these changes
Apr 29, 2026
Tianlei Wu (tianleiwu)
deleted the
copilot/update-ortvalue-inplace-cuda-graph
branch
April 29, 2026 21:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an
OrtValueoverload toupdate_inplaceso GPU-resident data can be copied directly to anotherOrtValuewithout roundtripping through CPU.onnxruntime_pybind_ortvalue.cc): Newupdate_inplace(const OrtValue*)overload. UsesCreateDataTransferMemCpyfor plugin EPs, with fallback to built-in copy functions for CUDA (including GPU↔GPU viaGetGPUDataTransfer()), MIGraphX, DML, and CANN.onnxruntime_inference_collection.py):update_inplacenow accepts either a numpy array or anOrtValue, dispatching to the appropriate C++ overload.onnxruntime_test_python_cudagraph.py): Covers CPU→CPU, GPU→GPU, CPU→GPU, and GPU→CPU OrtValue copy paths.Motivation and Context
CUDA graph replay requires inputs at fixed memory addresses. When source data (e.g., encoder output) is already on GPU, the only option was to use external libraries like
cuda-pythonfor device-to-device memcpy. This change makes that workflow native to ORT, per the approach suggested in the issue discussion: accept anOrtValueinupdate_inplaceto leverage ORT's existing data transfer infrastructure.