Fix QMoE blockwise quantization support for TRT-RTX execution provider#1926
Merged
Conversation
Contributor
Author
|
@kunal-vaishnavi @baijumeswani for review |
|
#1861 has broken model builder for TRT RTX EP for the cases where we use int4_block_size in olive recipe. We need the fix to that. |
- Remove bfloat16 scale conversion workaround (ORT 1.24 supports natively) - Fix zero_points: skip for TRT-RTX, always include for other EPs - Remove NvTensorRtRtx from internal EP checks (use 'trt-rtx' only) - Simplify make_qmoe_weights() to use int4_qmoe_block_size for all supported EPs (trt-rtx defaults to 128, cpu/webgpu default to 0)
Contributor
Author
|
@kunal-vaishnavi : i have addressed the issues, can u please have a look |
|
@thpereir FYI |
…aults - Rename int4_qmoe_block_size to qmoe_block_size (op supports int4 and int8) - Add CUDA to supported blockwise quantization EPs - Change default qmoe_block_size: 128 (trt-rtx), 32 (others) - Remove bfloat16 workarounds (ORT 1.24 supports natively) - Rename quant_attrs key from 'block_size' to 'qmoe_block_size'
anujj
force-pushed
the
gpt_oss_trt_rtx
branch
from
e8ddca5 to
588a3ce
Compare
Contributor
Author
|
Addressed the comments in the latest commit |
Contributor
|
Still going over the PR and reviewing it |
CUDA does not yet support block-wise quantization for QMoE
Contributor
Author
|
addressed @kunal-vaishnavi all comments |
thpereir
approved these changes
Jan 21, 2026
|
[like] Fernandes Crepaldi, Thiago reacted to your message:
…________________________________
From: Thiago Pereira Rocha ***@***.***>
Sent: Wednesday, January 21, 2026 9:32:28 PM
To: microsoft/onnxruntime-genai ***@***.***>
Cc: Fernandes Crepaldi, Thiago ***@***.***>; Comment ***@***.***>
Subject: Re: [microsoft/onnxruntime-genai] Fix QMoE blockwise quantization support for TRT-RTX execution provider (PR #1926)
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
@thpereir approved this pull request.
Code lgtm. Also run a quick gpt-oss regression and everything is working as expected
—
Reply to this email directly, view it on GitHub<#1926 (review)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABJXM4PND4G3WCVQEJIVZ3T4H7V6ZAVCNFSM6AAAAACPRLIRRSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTMOBZGQ4TANZZGU>.
You are receiving this because you commented.Message ID: ***@***.***>
|
kunal-vaishnavi
approved these changes
Jan 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add QMoE and BF16 support for TRT-RTX execution provider