[CUDA] Fix null allocator passed to plugin EP kernel PrePack - #29658
Merged
Conversation
Move the session-consuming assertions into the try/except else clause so static analysis can see they only run when InferenceSession succeeds.
…gin_ep_prepack_allocator
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a crash when ONNX Runtime is built with the CUDA EP as a plugin by ensuring the framework-provided pre-pack allocator is correctly forwarded through the EP-API kernel adapter into kernel PrePack() implementations.
Changes:
- Forward the incoming
OrtAllocator*throughKernelImpl::PrePackWeightImplinstead of dropping it and passing a nullAllocatorPtr. - Add a non-owning
IAllocatorWrappingOrtAllocator(OrtAllocator*)path so the adapter can wrap the framework-owned allocator without releasing it. - Add a Python regression test that constructs a
MatMulNBitsmodel intended to trigger weight pre-packing during session creation.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| include/onnxruntime/ep/adapter/op_kernel.h | Fixes allocator plumbing in plugin EP PrePackWeight adapter path. |
| include/onnxruntime/ep/adapter/allocator.h | Adds non-owning allocator wrapper constructor and routes calls via raw OrtAllocator* function pointers. |
| onnxruntime/test/python/transformers/test_cuda_plugin_ep.py | Adds regression test for CUDA plugin EP pre-pack allocator forwarding via MatMulNBits pre-pack. |
- allocator.h: correct the non-owning wrapper comment to state the PrePack allocator is provided/owned by the ORT framework/caller for the duration of the PrePack call. - test_cuda_plugin_ep.py: skip deterministically when compute capability < 7.5 (fpA_intB unsupported), force ORT_FPA_INTB_GEMM=1 so pre-packing runs, propagate session-creation exceptions instead of a catch-all skipTest, and only skip on known fpA_intB-unsupported runtime errors during sess.run.
Contributor
Author
|
@microsoft-github-policy-service rerun |
hariharans29
approved these changes
Jul 17, 2026
This was referenced Jul 17, 2026
tianleiwu
added a commit
that referenced
this pull request
Jul 18, 2026
This cherry-picks the following commits for the release: | Commit ID | PR Number | Commit Title | |-----------|-----------|-------------| | dd32f35 | #29590 | Fix libcudart.so.13 hard dependency in pybind module breaking import on CPU-only Linux | | cc44a4d | #29706 | [CUDA] Fix XQA GroupQueryAttention cudaErrorInvalidValue on Blackwell (sm_120) | | 23a7e9d | #29705 | [CUDA] Do not link nvrtc | | ee93f83 | #29711 | [CUDA] Update cuda arch list for packages of cuda 12.8 | | fea45a3 | #29620 | [CUDA] Add cuDNN-free ArgMax/ArgMin/ReduceSum and fix LogSoftmax on plugin EP | | f05b218 | #29624 | Enable Spectre-mitigated MSVC libs for BinSkim builds | | 1c89b86 | #29687 | [BUILD] CUDA_QUANT_PREPROCESS off by default and Adjust CI | | 41bd391 | #29658 | [CUDA] Fix null allocator passed to plugin EP kernel PrePack | | 405fbea | #28896 | Add Windows ARM64 CUDA plugin package and align CUDA metadata/artifact naming | | 308f24c | #29622 | Enable fpA_intB GEMM in CUDA builds and add configurable options | | 16ebc1d | #29731 | [Build] Use GPU pool to unblock CI temporarily | |5911a3a263| #29748 | Add OrtErrorCode::ORT_DEVICE_RESET | |6217f73ec5 | #29663 | Fix plugin EP allocator deleter lifetime | --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: GitHub Copilot <copilot@example.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com> Co-authored-by: Yen-Shi Wang <yenshiw@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When ONNX Runtime is built with the CUDA execution provider as a plugin
(
onnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON), the EP-API op-kernel adapter(
ep::adapter::KernelImpl::PrePackWeightImpl) received a validOrtAllocator*from the framework but discarded it and forwarded a null
AllocatorPtr{}into the wrapped kernel's
PrePack(). Any CUDA kernel that pre-packs a constantweight (e.g.
MatMulNBits,Conv,GroupQueryAttention, quantized MoE) thenallocated scratch through that null allocator and crashed during session
initialization:
This surfaced end to end as an ONNX Runtime GenAI
og.Model(...)failure on agpt-oss-20b (
MatMulNBits) model when the CUDA EP was loaded as a plugin: thetrivial init session succeeded, but the first real model session crashed while
pre-packing quantized weights.
Key Changes
include/onnxruntime/ep/adapter/op_kernel.hPrePackWeightImplnow wraps the incomingOrtAllocator*and forwards a validAllocatorPtrtoOpKernel::PrePackinstead of a nullAllocatorPtr{}.include/onnxruntime/ep/adapter/allocator.hIAllocatorWrappingOrtAllocator(OrtAllocator*)constructor. The framework owns the pre-pack allocator, so the wrapper must not take ownership (an owningOrt::Allocatorwould release it on destruction). Calls now go through the rawOrtAllocator*function pointers directly, preserving theReserve→Alloc(version ≥ 18) andGetStats/AllocOnStream(version ≥ 23) fallbacks.onnxruntime/test/python/transformers/test_cuda_plugin_ep.pytest_registration_matmul_nbits_prepack: builds a fp16MatMulNBitsmodel with a runtime-prepacked (weight_prepacked=0) quantized weight, so weight pre-packing (MatMulNBits::PrePack_B→IAllocator::MakeUniquePtr(alloc, ...)) runs during session creation. This crashed before the fix and now passes.Motivation and Context
The pre-pack allocator is provided and owned by the framework for the duration
of the
PrePackcall. The legacy (in-tree) CUDA EP received it correctly; onlythe plugin op-kernel adapter dropped it. The non-owning wrapper matches the
lifetime contract used elsewhere in the adapter (e.g.
KernelInfoGetAllocator)and keeps the CUDA-EP-as-plugin build behaviorally identical to the in-tree EP.
Testing
test_registration_matmul_nbits_prepackintest_cuda_plugin_ep.pypasses on a CUDA-EP-as-plugin build (
ORT_TEST_CUDA_PLUGIN_EP=1) and skipsgracefully when the device lacks fpA_intB GEMM support. It re-raises (fails)
if the
allocator != nullptrassertion recurs.end to end through the CUDA plugin EP (gpt-oss-20b,
MatMulNBits), including a100-sample MMLU sanity run.
test_cuda_plugin_ep.pyregistration tests continue to pass.