NvTensorRtRtx EP option in GenAI - model builder by BLSharda · Pull Request #1453 · microsoft/onnxruntime-genai

Bhavik Sharda (BLSharda) · 2025-05-06T13:56:01Z

Add NvTensorRtRtx EP option in GenAI - model builder

NvTensorRtRtx uses opset 21.
Enables use_qdq and uses Gelu standard op from opset 20.

…ensorRtRtx EP. - NvTensorRtRtx uses opset 21. - Enables use_qdq and uses Gelu standard op from opset 20.

Enabling Contrib-Ops Expansion Tested with Gemma-2b-it, Gemma-2-2b-it, deepseek llama and Nemotron 4b.

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>

Updating functions as hidden starts with _* Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>

Implemented separate _make_layernormalization for contirb-ops

Review comment fixed for Gelu parent function which decides between official and contrib-ops as well.

Fixed rebase conflict issues

Fixed more merge conflict issues

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>

Reverting unwanted changes

Bhavik Sharda (BLSharda) · 2025-05-14T16:45:48Z

kunal-vaishnavi, can you help retrigger pipeline? seems like "CUDNN" error not related to my change.

anshuman (anskumar01) · 2025-05-18T13:28:47Z

kunal-vaishnavi , can you please help retrigger the pipeline and help merge this change once review is final?

Bhavik Sharda (BLSharda) · 2025-05-19T12:08:08Z

kunal-vaishnavi , can you help merge this PR?

Bhavik Sharda (BLSharda) · 2025-05-20T16:00:39Z

Need help merge this PR as this is already reviewed & approved.

Also, the pipeline failure Linux CUDA x64 Build / linux-cuda-x64-build is unrelated to code changes in this PR.

+ az login --identity --username 63b63039-6328-442f-954b-5a64d124e5b4
ERROR: Passing the managed identity ID with --username is no longer supported.

Bhavik Sharda (BLSharda) · 2025-05-22T11:30:42Z

Need help merge this PR as this is already reviewed & approved.

Also, the pipeline failure Linux CUDA x64 Build / linux-cuda-x64-build is unrelated to code changes in this PR.
+ az login --identity --username 63b63039-6328-442f-954b-5a64d124e5b4
ERROR: Passing the managed identity ID with --username is no longer supported. 

kunal-vaishnavi and Baiju Meswani (@baijumeswani) , can you help merge this change?

Baiju Meswani (baijumeswani) · 2025-05-22T16:18:51Z

The issue was addressed in #1497. Please merge main and push.

Bhavik Sharda (BLSharda) · 2025-05-28T04:38:14Z

The issue was addressed in #1497. Please merge main and push.

Baiju Meswani (@baijumeswani), sure, after merge and push still hitting issue with same test "Linux CUDA x64 Build / linux-cuda-x64-build". And as this PR doesn't affect CUDA model builder, this is unrelated, right?

[E:onnxruntime:, cuda_call.cc:123 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=27d6b4d1345e ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=73 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyDeviceToHost, static_cast<cudaStream_t>(stream.GetHandle()));

Thanks, I see pipeline is passing now.

Adding option in Model builder to produce opset 21 ONNX model for NvT…

4ca87a5

…ensorRtRtx EP. - NvTensorRtRtx uses opset 21. - Enables use_qdq and uses Gelu standard op from opset 20.

Bhavik Sharda (BLSharda) marked this pull request as ready for review May 6, 2025 14:39

Bhavik Sharda (BLSharda) changed the title ~~Adding option in Model builder to produce opset 21 ONNX model for NvT…~~ NvTensorRtRtx EP option in GenAI - model builder May 6, 2025

Bhavik Sharda (BLSharda) added 2 commits May 7, 2025 14:37

Enabling Contrib-Ops Expansion

be5055d

Enabling Contrib-Ops Expansion Tested with Gemma-2b-it, Gemma-2-2b-it, deepseek llama and Nemotron 4b.

Merge branch 'main' into bhaviksharda/modelbuilder_ep_NvTensorRtRtx

51141d7

Nat Kershaw (MSFT) (natke) added the 0.8.0 label May 7, 2025