[TRTLLM-14880][feat] qualify Qwen3 dense for MX

docs/source/features/model-express.md

@@ -27,6 +27,7 @@ profiles:
 |---------|------------|-----------------|-------|----------|----------------------|-------------|
 | `llama-for-causal-lm-target-v1` | `LlamaForCausalLM` | `LlamaForCausalLM` / `llama` | Target model | 1 | `trtllm-llama-target-layout-v1` | Single-node dense BF16, unquantized weights and KV cache, TRTLLM attention, default fused RoPE, untied embeddings, TP=1 or 2, PP/CP=1, no LoRA, sparse attention, attention DP, speculative mode, or separately loaded draft model |
 | `qwen2-for-causal-lm-bf16-target-v1` | `Qwen2ForCausalLM` | `Qwen2ForCausalLM` / `qwen2` | Target model | 1 | `trtllm-qwen2-dense-target-layout-v1` | Single-node dense BF16, unquantized weights and KV cache, TRTLLM attention, default fused RoPE, untied embeddings, TP=1 or 2, PP/CP=1, no LoRA, sparse attention, attention DP, speculative mode, or separately loaded draft model |
+| `qwen3-for-causal-lm-bf16-target-v1` | `Qwen3ForCausalLM` | `Qwen3ForCausalLM` / `qwen3` | Target model | 1 | `trtllm-qwen3-dense-target-layout-v1` | Single-node dense BF16, unquantized weights and KV cache, TRTLLM attention, default fused QK-norm/RoPE, untied embeddings, TP=1 or 2, PP/CP=1, no LoRA, sparse attention, attention DP, speculative mode, or separately loaded draft model |
 
 The registry matches the exact root class, the architecture/model type captured
 from the resolved config before model construction, and any runtime constraints
@@ -57,14 +58,18 @@ standard checkpoint path. Target-plus-draft post-transform transfer remains
 disabled until layout state is tracked and qualified independently for each
 submodel.
 
-The Llama and Qwen2 profiles are text-only and do not enable reward-model, MoE,
-or vision-language roots. FP16, quantized weights or KV cache, alternate
-attention backends, YaRN or unfused RoPE, tied embeddings, TP greater than 2,
-PP greater than 1, CP greater than 1, LoRA, sparse attention, attention DP,
-multi-node transfer, and speculative decoding require separate qualification
-rows. The profiles do not constrain MoE-only backend and mapping settings
-because these dense roots do not consume them. `SourceIdentity` still requires
-donor and receiver configurations to match.
+The Llama, Qwen2, and Qwen3 profiles are text-only and do not enable
+reward-model, embedding, MoE, or vision-language roots. FP16, quantized weights
+or KV cache, alternate attention backends, YaRN, tied embeddings, TP greater
+than 2, PP greater than 1, CP greater than 1, LoRA, sparse attention, attention
+DP, multi-node transfer, and speculative decoding require separate
+qualification rows. Each profile also pins its qualified RoPE realization:
+Llama and Qwen2 require the default fused RoPE path, so unfused RoPE requires
+separate qualification for them, while Qwen3 fuses RoPE into the QK-norm
+kernel and therefore requires `rope_fusion=False` in the realized
+configuration. The profiles do not constrain MoE-only backend and mapping
+settings because these dense roots do not consume them. `SourceIdentity`
+still requires donor and receiver configurations to match.
 
 ### Adding a Model Family
 
@@ -112,12 +117,15 @@ pytest -v tests/integration/defs/model_express/test_model_express.py \
 
 Run the TP=2 rank-mapping qualification on four GPUs by selecting
 `llama-bf16-tp2`. `TRTLLM_MX_LLAMA_MODEL` can override the default TinyLlama
-checkpoint path. `TRTLLM_MX_E2E_REQUIRED=1` converts missing service, model,
-or NIXL prerequisites from skips into failures and must be set by a CI
-qualification stage. That stage must also allocate the GPUs declared by the
-selected test row. `TRTLLM_MX_E2E_TIMEOUT_S` controls the 1200-second timeout
-used for the baseline worker, receiver worker, and donor-readiness wait;
-increase it for slow model storage or startup.
+checkpoint path. The Qwen2 and Qwen3 profile rows use `qwen2-bf16-tp1` /
+`qwen2-bf16-tp2` and `qwen3-bf16-tp1` / `qwen3-bf16-tp2`, with optional model
+path overrides in `TRTLLM_MX_QWEN2_MODEL` and `TRTLLM_MX_QWEN3_MODEL`.
+`TRTLLM_MX_E2E_REQUIRED=1` converts missing service, model, or NIXL
+prerequisites from skips into failures and must be set by a CI qualification
+stage. That stage must also allocate the GPUs declared by the selected test
+row. `TRTLLM_MX_E2E_TIMEOUT_S` controls the 1200-second timeout used for the
+baseline worker, receiver worker, and donor-readiness wait; increase it for
+slow model storage or startup.
 
 The dedicated H100 CI stages own isolated Redis and ModelExpress 0.4.1
 sidecars. The two-GPU TP=1 stage is classified as multi-GPU: it runs
@@ -251,10 +259,10 @@ path.
 
 ## Notes and Limitations
 
-- Post-transform MX reception is currently limited to the exact Llama and
-  Qwen2/Qwen2.5 dense profiles above. Other roots and variants that do not
-  match the documented identity and runtime envelope safely fall back to
-  Hugging Face loading until explicitly qualified.
+- Post-transform MX reception is currently limited to the exact Llama,
+  Qwen2/Qwen2.5 dense, and Qwen3 dense profiles above. Other roots and variants
+  that do not match the documented identity and runtime envelope safely fall
+  back to Hugging Face loading until explicitly qualified.
 - The MX server and Redis lifecycle is external to TensorRT LLM. Every
   TensorRT LLM instance must be able to reach the configured MX server URL.
 - The MX server coordinates source discovery but does not store model weights.

tensorrt_llm/_torch/pyexecutor/model_loader.py

@@ -6,6 +6,7 @@
 import os
 import traceback
 import warnings
+from dataclasses import replace
 from enum import Enum
 from typing import Callable, Optional, Tuple
 
@@ -16,7 +17,8 @@
 from tensorrt_llm._torch.peft.lora.config import LoraConfig
 from tensorrt_llm._torch.weight_sharing import (
     LLAMA_POST_TRANSFORM_LAYOUT_ABI_V1,
-    QWEN2_DENSE_POST_TRANSFORM_LAYOUT_ABI_V1, ArtifactIdentity,
+    QWEN2_DENSE_POST_TRANSFORM_LAYOUT_ABI_V1,
+    QWEN3_DENSE_POST_TRANSFORM_LAYOUT_ABI_V1, ArtifactIdentity,
     IdentityCheckPolicy, PostTransformConfigIdentity, PostTransformFeature,
     PostTransformProfile, PostTransformProfileRegistry,
     PostTransformQualificationDecision, PostTransformRuntimeConfig,
@@ -79,6 +81,10 @@
     rope_types=frozenset({"default"}),
     rope_fusion=frozenset({True}),
 )
+_MX_QWEN3_BF16_DENSE_RUNTIME_CONSTRAINTS = replace(
+    _MX_BF16_DENSE_RUNTIME_CONSTRAINTS,
+    rope_fusion=frozenset({False}),
+)
 
 
 def _validate_and_adjust_mamba_snapshot_config(config: ModelConfig,
@@ -390,6 +396,7 @@ def _post_transform_profile_registry(cls) -> PostTransformProfileRegistry:
         if cls._POST_TRANSFORM_PROFILE_REGISTRY is None:
             from ..models.modeling_llama import LlamaForCausalLM
             from ..models.modeling_qwen import Qwen2ForCausalLM
+            from ..models.modeling_qwen3 import Qwen3ForCausalLM
             cls._POST_TRANSFORM_PROFILE_REGISTRY = PostTransformProfileRegistry(
                 profiles=(
                     PostTransformProfile(
@@ -417,6 +424,20 @@ def _post_transform_profile_registry(cls) -> PostTransformProfileRegistry:
                         transfer_scope=PostTransformTransferScope.TARGET_MODEL,
                         runtime_constraints=_MX_BF16_DENSE_RUNTIME_CONSTRAINTS,
                     ),
+                    PostTransformProfile(
+                        profile_id="qwen3-for-causal-lm-bf16-target-v1",
+                        root_model_class=Qwen3ForCausalLM,
+                        architecture="Qwen3ForCausalLM",
+                        model_type="qwen3",
+                        speculative_mode=None,
+                        protocol_version=cls.
+                        _MX_STAGED_RECEIVER_TRANSFORM_PROTOCOL_VERSION,
+                        transform_abi_id=
+                        QWEN3_DENSE_POST_TRANSFORM_LAYOUT_ABI_V1,
+                        transfer_scope=PostTransformTransferScope.TARGET_MODEL,
+                        runtime_constraints=
+                        _MX_QWEN3_BF16_DENSE_RUNTIME_CONSTRAINTS,
+                    ),
                 ))
         return cls._POST_TRANSFORM_PROFILE_REGISTRY
 

tensorrt_llm/_torch/weight_sharing/__init__.py

@@ -21,6 +21,7 @@
 from tensorrt_llm._torch.weight_sharing.post_transform_profiles import (
     LLAMA_POST_TRANSFORM_LAYOUT_ABI_V1,
     QWEN2_DENSE_POST_TRANSFORM_LAYOUT_ABI_V1,
+    QWEN3_DENSE_POST_TRANSFORM_LAYOUT_ABI_V1,
     PostTransformConfigIdentity,
     PostTransformFeature,
     PostTransformProfile,
@@ -43,10 +44,14 @@
 
 __all__ = [
     "ARTIFACT_IDENTITY_FORMAT_VERSION",
-    "ArtifactIdentity",
     "LLAMA_POST_TRANSFORM_LAYOUT_ABI_V1",
     "QWEN2_DENSE_POST_TRANSFORM_LAYOUT_ABI_V1",
+    "QWEN3_DENSE_POST_TRANSFORM_LAYOUT_ABI_V1",
     "SOURCE_IDENTITY_FORMAT_VERSION",
+    "ArtifactIdentity",
+    "IdentityCheckDecision",
+    "IdentityCheckPolicy",
+    "IdentityMatchResult",
     "PostTransformConfigIdentity",
     "PostTransformFeature",
     "PostTransformProfile",
@@ -57,9 +62,6 @@
     "PostTransformRuntimeConstraints",
     "PostTransformTransferScope",
     "SourceIdentity",
-    "IdentityMatchResult",
-    "IdentityCheckPolicy",
-    "IdentityCheckDecision",
     "SourceIdentityMismatchError",
     "check_weight_sharing_compatibility",
 ]

tensorrt_llm/_torch/weight_sharing/post_transform_profiles.py

@@ -42,6 +42,9 @@
 # Stable contract for unquantized Qwen2 dense fused-QKV and fused-gate-up
 # tensors plus the target-only receiver finalization used by its first profile.
 QWEN2_DENSE_POST_TRANSFORM_LAYOUT_ABI_V1 = "trtllm-qwen2-dense-target-layout-v1"
+# Stable contract for unquantized Qwen3 dense fused-QKV and fused-gate-up
+# tensors, Q/K norm state, and target-only receiver finalization.
+QWEN3_DENSE_POST_TRANSFORM_LAYOUT_ABI_V1 = "trtllm-qwen3-dense-target-layout-v1"
 _MISSING = object()
 
 

tests/integration/defs/model_express/test_model_express.py

@@ -116,6 +116,46 @@ class MxE2ECase:
         id="llama-bf16-tp2",
         marks=pytest.mark.skip_less_device(4),
     ),
+    pytest.param(
+        MxE2ECase(
+            model_env="TRTLLM_MX_QWEN2_MODEL",
+            default_model_subdir="Qwen2-7B-Instruct",
+            repository_cache_prefix="models--trtllm-mx-e2e--qwen2-tp1",
+            tp_size=1,
+        ),
+        id="qwen2-bf16-tp1",
+        marks=pytest.mark.skip_less_device(2),
+    ),
+    pytest.param(
+        MxE2ECase(
+            model_env="TRTLLM_MX_QWEN2_MODEL",
+            default_model_subdir="Qwen2-7B-Instruct",
+            repository_cache_prefix="models--trtllm-mx-e2e--qwen2-tp2",
+            tp_size=2,
+        ),
+        id="qwen2-bf16-tp2",
+        marks=pytest.mark.skip_less_device(4),
+    ),
+    pytest.param(
+        MxE2ECase(
+            model_env="TRTLLM_MX_QWEN3_MODEL",
+            default_model_subdir="Qwen3/Qwen3-8B",
+            repository_cache_prefix="models--trtllm-mx-e2e--qwen3-tp1",
+            tp_size=1,
+        ),
+        id="qwen3-bf16-tp1",
+        marks=pytest.mark.skip_less_device(2),
+    ),
+    pytest.param(
+        MxE2ECase(
+            model_env="TRTLLM_MX_QWEN3_MODEL",
+            default_model_subdir="Qwen3/Qwen3-8B",
+            repository_cache_prefix="models--trtllm-mx-e2e--qwen3-tp2",
+            tp_size=2,
+        ),
+        id="qwen3-bf16-tp2",
+        marks=pytest.mark.skip_less_device(4),
+    ),
 )
 
 

tests/integration/test_lists/test-db/l0_model_express.yml

@@ -19,6 +19,8 @@ l0_model_express:
       orchestrator: mpi
   tests:
   - model_express/test_model_express.py::test_mx_donor_receiver[llama-bf16-tp1]
+  - model_express/test_model_express.py::test_mx_donor_receiver[qwen2-bf16-tp1]
+  - model_express/test_model_express.py::test_mx_donor_receiver[qwen3-bf16-tp1]
 - condition:
     ranges:
       system_gpu_count:
@@ -35,3 +37,5 @@ l0_model_express:
       orchestrator: mpi
   tests:
   - model_express/test_model_express.py::test_mx_donor_receiver[llama-bf16-tp2]
+  - model_express/test_model_express.py::test_mx_donor_receiver[qwen2-bf16-tp2]
+  - model_express/test_model_express.py::test_mx_donor_receiver[qwen3-bf16-tp2]