Skip to content

Enable Qwen3.5 35B-A3B MoE text export for TRT-RTX EP - #2186

Merged
baijumeswani merged 1 commit into
microsoft:mainfrom
yen-shi:yenshiw/trt-rtx-qwen35-35b-moe
May 29, 2026
Merged

Enable Qwen3.5 35B-A3B MoE text export for TRT-RTX EP#2186
baijumeswani merged 1 commit into
microsoft:mainfrom
yen-shi:yenshiw/trt-rtx-qwen35-35b-moe

Conversation

@yen-shi

@yen-shi yen-shi commented May 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Enable Qwen3.5 35B-A3B MoE text-only export and inference with the TRT-RTX execution provider.

  • Emit a decoder-only GenAI model type for TRT-RTX text-only Qwen3.5-MoE exports.
  • Keep the existing qwen3_5_moe VLM path intact for non-TRT-RTX/non-text behavior.
  • Preserve model-provided swiglu_limit when available, and fall back to +inf when the config omits it.
  • Allow Python examples to use the GenAI provider name NvTensorRtRtx while registering the external ORT EP library as NvTensorRTRTXExecutionProvider.

Validation

  • Exported Qwen3.5 35B-A3B MoE INT4 QDQ text-only model with TRT-RTX EP.
  • Verified generated config:
    • model.type = qwen3_5_moe_text
    • decoder file is text.onnx
    • TRT-RTX provider options include enable_cuda_graph = 1
  • Verified exported ONNX:
    • QMoE node count: 40
    • no QMoE nodes missing swiglu_limit
  • Ran TRT-RTX EP inference with model-qa.py using -e NvTensorRtRtx.
  • Smoke-tested the same -e NvTensorRtRtx alias on Qwen3.5 0.8B and 9B TRT-RTX packages.

Notes

  • This change does not remove or alter the existing qwen3_5_moe VLM model registration.
  • The TRT-RTX text-only path is selected only for the TRT-RTX export path with text-only settings.
  • The Python example alias is intentionally minimal: it only maps the GenAI provider name to the external ORT EP library name during registration.

Copilot AI review requested due to automatic review settings May 27, 2026 01:42
@yen-shi
yen-shi requested a review from a team as a code owner May 27, 2026 01:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Updates Qwen3.5-MoE builder logic and EP registration to better support TRT-RTX text-only execution, and adds a new model type identifier for LLM classification.

Changes:

  • Select a different Qwen3.5-MoE model_type when running TRT-RTX with text-only.
  • Populate swiglu_limit in MoE attributes with a fallback lookup/default.
  • Add qwen3_5_moe_text to the C++ LLM model type allowlist and accept NvTensorRtRtx as an EP alias in the Python examples.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
src/python/py/models/builders/qwen.py Adjusts Qwen3.5-MoE model type selection for TRT-RTX text-only and adds swiglu_limit handling.
src/models/model_type.h Adds qwen3_5_moe_text to the LLM classification list.
examples/python/common.py Adds NvTensorRtRtx EP option and maps it to the provider library name when registering.

Comment thread src/python/py/models/builders/qwen.py
Comment thread src/python/py/models/builders/qwen.py Outdated
Comment thread src/models/model_type.h Outdated
Comment thread src/python/py/models/builders/qwen.py Fixed
@anskumar01

Copy link
Copy Markdown

@kunal-vaishnavi , @baijumeswani , can you please review

@yen-shi
yen-shi force-pushed the yenshiw/trt-rtx-qwen35-35b-moe branch 4 times, most recently from a824e5d to cebee60 Compare May 27, 2026 07:10
Comment thread examples/python/common.py Outdated
Comment thread src/python/py/models/builders/qwen.py
Comment thread examples/python/common.py Outdated
Comment thread src/python/py/models/builders/qwen.py Outdated
Comment thread src/models/model_type.h Outdated
@yen-shi
yen-shi force-pushed the yenshiw/trt-rtx-qwen35-35b-moe branch 2 times, most recently from 5016191 to fff32ae Compare May 28, 2026 07:24
Comment thread src/python/py/models/builders/qwen.py Dismissed
@yen-shi
yen-shi force-pushed the yenshiw/trt-rtx-qwen35-35b-moe branch from fff32ae to 0257321 Compare May 28, 2026 08:01
@yen-shi

yen-shi commented May 29, 2026

Copy link
Copy Markdown
Contributor Author

Hi @kunal-vaishnavi,
Do you know how to pass or restart the pipeline?
One windows pipeline failed, but I don't see clear error message from the logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants