Skip to content

[CUDA] Fix null allocator passed to plugin EP kernel PrePack - #29658

Merged
tianleiwu merged 6 commits into
mainfrom
tlwu/20260710/fix_plugin_ep_prepack_allocator
Jul 17, 2026
Merged

[CUDA] Fix null allocator passed to plugin EP kernel PrePack#29658
tianleiwu merged 6 commits into
mainfrom
tlwu/20260710/fix_plugin_ep_prepack_allocator

Conversation

@tianleiwu

Copy link
Copy Markdown
Contributor

Description

When ONNX Runtime is built with the CUDA execution provider as a plugin
(onnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON), the EP-API op-kernel adapter
(ep::adapter::KernelImpl::PrePackWeightImpl) received a valid OrtAllocator*
from the framework but discarded it and forwarded a null AllocatorPtr{}
into the wrapped kernel's PrePack(). Any CUDA kernel that pre-packs a constant
weight (e.g. MatMulNBits, Conv, GroupQueryAttention, quantized MoE) then
allocated scratch through that null allocator and crashed during session
initialization:

IAllocator::ValidateAllocator(const T&) [with T = std::shared_ptr<onnxruntime::IAllocator>]
allocator != nullptr was false

This surfaced end to end as an ONNX Runtime GenAI og.Model(...) failure on a
gpt-oss-20b (MatMulNBits) model when the CUDA EP was loaded as a plugin: the
trivial init session succeeded, but the first real model session crashed while
pre-packing quantized weights.

Key Changes

File Change
include/onnxruntime/ep/adapter/op_kernel.h PrePackWeightImpl now wraps the incoming OrtAllocator* and forwards a valid AllocatorPtr to OpKernel::PrePack instead of a null AllocatorPtr{}.
include/onnxruntime/ep/adapter/allocator.h Add a non-owning IAllocatorWrappingOrtAllocator(OrtAllocator*) constructor. The framework owns the pre-pack allocator, so the wrapper must not take ownership (an owning Ort::Allocator would release it on destruction). Calls now go through the raw OrtAllocator* function pointers directly, preserving the ReserveAlloc (version ≥ 18) and GetStats/AllocOnStream (version ≥ 23) fallbacks.
onnxruntime/test/python/transformers/test_cuda_plugin_ep.py Add test_registration_matmul_nbits_prepack: builds a fp16 MatMulNBits model with a runtime-prepacked (weight_prepacked=0) quantized weight, so weight pre-packing (MatMulNBits::PrePack_BIAllocator::MakeUniquePtr(alloc, ...)) runs during session creation. This crashed before the fix and now passes.

Motivation and Context

The pre-pack allocator is provided and owned by the framework for the duration
of the PrePack call. The legacy (in-tree) CUDA EP received it correctly; only
the plugin op-kernel adapter dropped it. The non-owning wrapper matches the
lifetime contract used elsewhere in the adapter (e.g. KernelInfoGetAllocator)
and keeps the CUDA-EP-as-plugin build behaviorally identical to the in-tree EP.

Testing

  • New test_registration_matmul_nbits_prepack in test_cuda_plugin_ep.py
    passes on a CUDA-EP-as-plugin build (ORT_TEST_CUDA_PLUGIN_EP=1) and skips
    gracefully when the device lacks fpA_intB GEMM support. It re-raises (fails)
    if the allocator != nullptr assertion recurs.
  • Verified the model that originally reproduced the crash now creates and runs
    end to end through the CUDA plugin EP (gpt-oss-20b, MatMulNBits), including a
    100-sample MMLU sanity run.
  • Existing test_cuda_plugin_ep.py registration tests continue to pass.

Comment thread onnxruntime/test/python/transformers/test_cuda_plugin_ep.py Fixed
Move the session-consuming assertions into the try/except else clause so
static analysis can see they only run when InferenceSession succeeds.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a crash when ONNX Runtime is built with the CUDA EP as a plugin by ensuring the framework-provided pre-pack allocator is correctly forwarded through the EP-API kernel adapter into kernel PrePack() implementations.

Changes:

  • Forward the incoming OrtAllocator* through KernelImpl::PrePackWeightImpl instead of dropping it and passing a null AllocatorPtr.
  • Add a non-owning IAllocatorWrappingOrtAllocator(OrtAllocator*) path so the adapter can wrap the framework-owned allocator without releasing it.
  • Add a Python regression test that constructs a MatMulNBits model intended to trigger weight pre-packing during session creation.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File Description
include/onnxruntime/ep/adapter/op_kernel.h Fixes allocator plumbing in plugin EP PrePackWeight adapter path.
include/onnxruntime/ep/adapter/allocator.h Adds non-owning allocator wrapper constructor and routes calls via raw OrtAllocator* function pointers.
onnxruntime/test/python/transformers/test_cuda_plugin_ep.py Adds regression test for CUDA plugin EP pre-pack allocator forwarding via MatMulNBits pre-pack.

Comment thread include/onnxruntime/ep/adapter/allocator.h
Comment thread onnxruntime/test/python/transformers/test_cuda_plugin_ep.py
Comment thread onnxruntime/test/python/transformers/test_cuda_plugin_ep.py Outdated
Comment thread onnxruntime/test/python/transformers/test_cuda_plugin_ep.py
- allocator.h: correct the non-owning wrapper comment to state the PrePack
  allocator is provided/owned by the ORT framework/caller for the duration
  of the PrePack call.
- test_cuda_plugin_ep.py: skip deterministically when compute capability <
  7.5 (fpA_intB unsupported), force ORT_FPA_INTB_GEMM=1 so pre-packing runs,
  propagate session-creation exceptions instead of a catch-all skipTest, and
  only skip on known fpA_intB-unsupported runtime errors during sess.run.
@tianleiwu

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service rerun

@tianleiwu
tianleiwu merged commit 41bd391 into main Jul 17, 2026
86 of 95 checks passed
@tianleiwu
tianleiwu deleted the tlwu/20260710/fix_plugin_ep_prepack_allocator branch July 17, 2026 00:59
tianleiwu added a commit that referenced this pull request Jul 18, 2026
This cherry-picks the following commits for the release:

| Commit ID | PR Number | Commit Title |
|-----------|-----------|-------------|
| dd32f35 | #29590 | Fix libcudart.so.13 hard dependency in pybind
module breaking import on CPU-only Linux |
| cc44a4d | #29706 | [CUDA] Fix XQA GroupQueryAttention
cudaErrorInvalidValue on Blackwell (sm_120) |
| 23a7e9d | #29705 | [CUDA] Do not link nvrtc |
| ee93f83 | #29711 | [CUDA] Update cuda arch list for packages of
cuda 12.8 |
| fea45a3 | #29620 | [CUDA] Add cuDNN-free ArgMax/ArgMin/ReduceSum
and fix LogSoftmax on plugin EP |
| f05b218 | #29624 | Enable Spectre-mitigated MSVC libs for BinSkim
builds |
| 1c89b86 | #29687 | [BUILD] CUDA_QUANT_PREPROCESS off by default and
Adjust CI |
| 41bd391 | #29658 | [CUDA] Fix null allocator passed to plugin EP
kernel PrePack |
| 405fbea | #28896 | Add Windows ARM64 CUDA plugin package and align
CUDA metadata/artifact naming |
| 308f24c | #29622 | Enable fpA_intB GEMM in CUDA builds and add
configurable options |
| 16ebc1d | #29731 | [Build] Use GPU pool to unblock CI temporarily |
|5911a3a263| #29748 | Add OrtErrorCode::ORT_DEVICE_RESET |
|6217f73ec5 | #29663 | Fix plugin EP allocator deleter lifetime |

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: GitHub Copilot <copilot@example.com>
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
Co-authored-by: Yen-Shi Wang <yenshiw@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants