[TRTLLM-8958][feat] and [TRTLLM-8960]: create ConfigurableMoE and support TRTLLMGenFusedMoE as backend #9486
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xxi-nv
requested review from
HuiGao-NV,
QiJune,
dongxuy04,
nekorobov,
symphonylyh and
syuoni
Contributor
📝 WalkthroughWalkthroughThis pull request refactors the MoE framework with a new composition-based ConfigurableMoE class, communication strategy updates renaming MNNVL backends to NVLink terminology, and adjustments to weight loading logic for MoE modules. Changes introduce lazy load balancer initialization via Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Areas requiring extra attention:
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Tip 📝 Customizable high-level summaries are now available in beta!You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.
Example instruction:
Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_deepgemm.py (1)
1-1: Add NVIDIA copyright header as per project guidelinesThis Python source file appears to be missing the standard NVIDIA copyright header with the current year at the top. To align with the TensorRT‑LLM coding guidelines for
*.pyfiles, please add the appropriate header comment above the imports (you can copy the exact format from nearby MoE Python modules in this repository).
🧹 Nitpick comments (17)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_deepgemm.py (1)
384-396: Clarify/configure moe_max_num_tokens capping and preserve ModelConfig’s frozen stateThe DeepGemm‑specific cap itself makes sense for OOM mitigation, but a couple of details are worth tightening:
18688is a backend‑specific magic number; today it’s only documented in comments and duplicated in code. If this formula ever changes, comments and code can silently diverge. Consider promoting this to a named constant (module‑ or class‑level) so the cap is defined in one place and the intent is clearer.- This path mutates the passed‑in
model_configand unconditionally sets_frozen = Trueafter clamping. If a caller passes a sharedModelConfig(e.g., reused across multiple layers/backends) or intentionally keeps it unfrozen, this will globally change its state. A lighter‑weight pattern is to preserve and restore the previous_frozenvalue while still enforcing the cap:- default_moe_max_num_tokens = 18688 - if model_config.moe_max_num_tokens > default_moe_max_num_tokens: - model_config._frozen = False - model_config.moe_max_num_tokens = default_moe_max_num_tokens - model_config._frozen = True + default_moe_max_num_tokens = 18688 # consider promoting to a named constant + if model_config.moe_max_num_tokens > default_moe_max_num_tokens: + prev_frozen = getattr(model_config, "_frozen", False) + model_config._frozen = False + model_config.moe_max_num_tokens = default_moe_max_num_tokens + model_config._frozen = prev_frozenYou may also want to confirm that clamping
model_config.moe_max_num_tokensat the config level (vs. using a DeepGemm‑local limit) is the desired behavior for other consumers that might share this config instance.tensorrt_llm/_torch/models/modeling_deepseekv3.py (1)
58-60: ConfigurableMoE backend handling in DeepseekV3 loader is sound (consider centralizing helper)The new
elif names[-1] == "backend" and isinstance(module, MoE)branch correctly reuses the parent prefix, applies the samedown_proj/up_proj/gate_proj→w2/w3/w1renaming as the existing"experts"path, and then delegates to the backend’sload_weights. Together with thelen(module._parameters) <= 0guard, this cleanly separates legacy direct‑MoE (experts) from ConfigurableMoE backend loading.Given similar
.backendhandling now exists in multiple modeling files, you might eventually factor out a tiny utility (e.g., a helper inmodeling_utilsthat normalizes MoE module names for weight filtering) to keep this logic in one place, but that’s optional.Also applies to: 385-399
tensorrt_llm/_torch/models/modeling_utils.py (1)
865-876: ConfigurableMoE.backendname normalization looks correct; consider de-duplicating helper.The new
if names[-1] == "backend" and isinstance(module, MoE)block in both loaders is a good, minimal way to reconcile saved MoE weight keys (without.backend) with the newConfigurableMoE.backendmodule hierarchy. It’s scoped toMoEinstances, so other modules namedbackendare not affected, and it integrates cleanly with the existingparams_mapandfilter_weightslogic.If this pattern needs to be reused further, consider extracting a small helper (e.g.,
normalize_moe_module_name(name, module)) to avoid duplicating the same condition in_load_weights_impland_load_weights_impl_v2, but that’s optional and not required for correctness.Also applies to: 969-980
tensorrt_llm/_torch/modules/fused_moe/communication/__init__.py (1)
21-30: NVLink comm rename is clear; consider a compat alias if users relied on old MNNVL names.Switching exports to
NVLinkTwoSidedandNVLinkOneSidedand updating the docstring/__all__is straightforward. If any external code importsMnnvlLatencyorMNNVLThroughputfrom this package, those imports will now fail; if that API was considered public, you may want a small alias layer in this module to preserve backward compatibility.Also applies to: 37-48
tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.py (1)
16-22: NVLinkTwoSided implementation is consistent; you can optionally silence Ruff ARG002 with underscore-prefixed args.The NVLINK two-sided class correctly aligns its docstrings, caches
_is_platform_supportedin__init__, and uses a staticis_platform_supported()consistent with other comm backends. Returningself._is_platform_supportedfromis_workload_feasibleis fine for now, even if you don’t yet useall_rank_num_tokens/num_chunks.If you want to address Ruff’s unused-argument warnings (ARG002) without changing behavior, you can rename the parameters e.g.:
- def is_workload_feasible(self, all_rank_num_tokens: List[int], num_chunks: int) -> bool: + def is_workload_feasible(self, _all_rank_num_tokens: List[int], _num_chunks: int) -> bool:The TRY003 warning on the ValueError message is purely stylistic; I’d treat adjusting that as optional.
Also applies to: 35-39, 63-67, 71-80, 87-95, 139-147, 184-199
tests/unittest/_torch/modules/test_fused_moe.py (5)
19-20: PretrainedConfig + ModelConfig wiring fortest_fused_moelooks correct; helper could reduce duplication.Creating a bare
PretrainedConfigand populatingnum_experts,hidden_size,intermediate_size, andtorch_dtypebefore passing it intoModelConfig(pretrained_config=..., mapping=..., moe_backend=...)matches the local test hyperparameters and is a reasonable way to satisfy the new backend requirements.Given this pattern repeats in several tests, consider a small helper like
build_test_pretrained_config(num_experts, hidden_size, intermediate_size, dtype)to centralize the boilerplate and keep future changes to required fields in one place.Also applies to: 144-158
599-613: FP8 fused MoE test’s PretrainedConfig usage is consistent with the base test.In
test_fused_moe_fp8, thePretrainedConfigfields mirror the localNUM_EXPERTS,HIDDEN_SIZE,INTERMEDIATE_SIZE, anddtype, and feeding it intoModelConfig(pretrained_config=..., quant_config=..., moe_backend=...)is aligned with the new MoE backend expectations. No functional issues spotted here.
1456-1468: NVFP4 fused MoE test correctly provides pretrained_config to ModelConfig.For
test_fused_moe_nvfp4, the constructedPretrainedConfigmatches the NVFP4 test’sNUM_EXPERTS,HIDDEN_SIZE,INTERMEDIATE_SIZE, anddtype, and is passed intoModelConfig(pretrained_config=..., quant_config=quant_config, moe_backend=moe_backend)before callingcreate_moe. This should give ConfigurableMoE/TRTLLM backends the needed metadata without affecting the reference path.
1950-1963: MXFP4/MXFP8 fused MoE test also wires pretrained_config correctly.In
test_fused_moe_mxfp4_mxfp8, thePretrainedConfigis constructed withNUM_EXPERTS,HIDDEN_SIZE_UNPADDED,INTERMEDIATE_SIZE_UNPADDED, anddtype, and then passed intoModelConfig(pretrained_config=..., quant_config=quant_config, moe_backend=moe_backend)forcreate_moe. This matches the quantization setup and hidden-size padding logic used later in the test.
2258-2271: W4A16_MXFP4 fused MoE test’s pretrained_config setup is consistent with other NVLink/quant tests.For
test_fused_moe_w4a8_nvfp4_fp8, thePretrainedConfigfields are aligned with the localNUM_EXPERTS,HIDDEN_SIZE,INTERMEDIATE_SIZE, anddtype, and the resultingModelConfig(pretrained_config=..., quant_config=quant_config, moe_backend=moe_backend)is used only for the fused MoE path. This is consistent with the other quantized MoE tests and should integrate cleanly with the new backend selection logic.tensorrt_llm/_torch/modules/fused_moe/create_moe.py (1)
108-125: Duplicate parameter inference logic increate_moe_backendandcreate_moe.The parameter inference from
pretrained_config(lines 108-125 and 308-325) is duplicated. Sincecreate_moealways callscreate_moe_backend, the inference increate_moe_backend(lines 108-125) is sufficient. The code increate_moe(lines 308-325) can be simplified.Consider removing the duplicate inference in
create_moesincecreate_moe_backendalready handles it:def create_moe( routing_method: BaseMoeRoutingMethod, num_experts: Optional[int] = None, hidden_size: Optional[int] = None, intermediate_size: Optional[int] = None, dtype: Optional[torch.dtype] = None, ... ) -> MoE: - # Get parameters from pretrained_config if not explicitly provided - pretrained_config = model_config.pretrained_config - if num_experts is None: - assert pretrained_config is not None, "num_experts must be provided or model_config.pretrained_config must be set" - num_experts = pretrained_config.num_experts - if hidden_size is None: - assert pretrained_config is not None, "hidden_size must be provided or model_config.pretrained_config must be set" - hidden_size = pretrained_config.hidden_size - if intermediate_size is None: - assert pretrained_config is not None, "intermediate_size must be provided or model_config.pretrained_config must be set" - # For MoE models, prefer moe_intermediate_size if available - if hasattr(pretrained_config, 'moe_intermediate_size'): - intermediate_size = pretrained_config.moe_intermediate_size - else: - intermediate_size = pretrained_config.intermediate_size - if dtype is None and pretrained_config is not None and hasattr( - pretrained_config, 'torch_dtype'): - dtype = pretrained_config.torch_dtype - moe_cls = get_moe_cls(model_config, override_quant_config) + # Parameters will be inferred from pretrained_config in create_moe_backend if NoneHowever, this refactor may require adjusting the
ConfigurableMoEpath at lines 336-352 to handleNonevalues, so it can be deferred.Also applies to: 308-325
tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py (2)
561-561: Remove unused variable assignments.The static analysis correctly identifies that
mxfp8_xandsfare assigned but never used. These appear to be leftover from previous code.Apply this diff to remove the unused assignments:
- mxfp8_x, sf = x, x_sf
300-303: Unused variablex_colin w4a8_mxfp4_mxfp8 quantization path.
x_colis assigned but never used. Consider removing the assignment if not needed.elif self.has_w4a8_mxfp4_mxfp8: x, x_sf = torch.ops.trtllm.mxfp8_quantize( x, False, alignment=self.quant_method.input_hidden_alignment) - x_row, x_col = x.shape[0], x.shape[1] + x_row = x.shape[0]tensorrt_llm/_torch/modules/fused_moe/configurable_moe.py (4)
101-101: Mutable default argumentModelConfig()in function signature.Using a mutable object as a default argument can lead to unexpected behavior if the object is modified. While
ModelConfig()is likely a dataclass that won't be mutated, this is a best practice concern flagged by the linter.Consider using
Noneas default and creating the instance inside the function:- model_config: ModelConfig = ModelConfig(), + model_config: Optional[ModelConfig] = None,Then at the start of
__init__:if model_config is None: model_config = ModelConfig()
229-232: Consider usingValueErrorinstead ofassertfor configuration validation.Assertions can be disabled with
python -O, which would skip this validation in optimized mode. For configuration validation that should always run, consider using explicit exceptions.if self.apply_router_weight_on_input: - assert self.routing_method.top_k == 1, ( - "apply_router_weight_on_input only supports top-1 routing" - ) + if self.routing_method.top_k != 1: + raise ValueError("apply_router_weight_on_input only supports top-1 routing")
724-724: Consider addingstrict=Truetozip()for safety.Adding
strict=Truewould catch mismatched list lengths, providing an early error ifx_listandrouter_logits_listhave different sizes.- for idx_chunk, (x_chunk, router_logits_chunk) in enumerate(zip(x_list, router_logits_list)): + for idx_chunk, (x_chunk, router_logits_chunk) in enumerate(zip(x_list, router_logits_list, strict=True)):
234-245: Unusedmodel_configparameter in_create_comm_strategy.This method always returns
Nonefor lazy creation, making themodel_configparameter unused. Consider documenting this or removing the parameter if lazy creation is the intended pattern.If the parameter is for future use, add a comment:
def _create_comm_strategy(self, model_config: ModelConfig) -> Optional[Communication]: """ Create communication strategy based on configuration Default: None (will use factory to auto-select when needed) Auto-selects best strategy based on hardware and configuration """ + # Note: model_config is reserved for future use when non-lazy creation is needed # Communication strategy is None by default # Will be created lazily in determine_communication_method() when first needed # For now, return None and create on-demand return None
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (21)
tensorrt_llm/_torch/model_config.py(1 hunks)tensorrt_llm/_torch/models/modeling_deepseekv3.py(2 hunks)tensorrt_llm/_torch/models/modeling_gpt_oss.py(1 hunks)tensorrt_llm/_torch/models/modeling_hunyuan_moe.py(2 hunks)tensorrt_llm/_torch/models/modeling_utils.py(2 hunks)tensorrt_llm/_torch/modules/fused_moe/communication/__init__.py(2 hunks)tensorrt_llm/_torch/modules/fused_moe/communication/base.py(1 hunks)tensorrt_llm/_torch/modules/fused_moe/communication/communication_factory.py(8 hunks)tensorrt_llm/_torch/modules/fused_moe/communication/deep_ep.py(2 hunks)tensorrt_llm/_torch/modules/fused_moe/communication/deep_ep_low_latency.py(2 hunks)tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.py(13 hunks)tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.py(6 hunks)tensorrt_llm/_torch/modules/fused_moe/configurable_moe.py(1 hunks)tensorrt_llm/_torch/modules/fused_moe/create_moe.py(8 hunks)tensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.py(5 hunks)tensorrt_llm/_torch/modules/fused_moe/fused_moe_deepgemm.py(1 hunks)tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py(20 hunks)tensorrt_llm/_torch/modules/fused_moe/fused_moe_vanilla.py(1 hunks)tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py(1 hunks)tensorrt_llm/_torch/modules/fused_moe/interface.py(5 hunks)tests/unittest/_torch/modules/test_fused_moe.py(6 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (CODING_GUIDELINES.md)
**/*.py: The code developed for TensorRT-LLM should conform to Python 3.8+
Indent Python code with 4 spaces; do not use tabs
Always maintain the namespace when importing in Python, even if only one class or function from a module is used (e.g., usefrom package.subpackage import fooand thenfoo.SomeClass()instead offrom package.subpackage.foo import SomeClass)
Python filenames should use snake_case (e.g.,some_file.py)
Python class names should use PascalCase (e.g.,class SomeClass)
Python function and method names should use snake_case (e.g.,def my_awesome_function():)
Python local variable names should use snake_case, with prefixkfor variable names that start with a number (e.g.,k_99th_percentile = ...)
Python global variables should use upper snake_case with prefixG(e.g.,G_MY_GLOBAL = ...)
Python constants should use upper snake_case (e.g.,MY_CONSTANT = ...)
Avoid shadowing variables declared in an outer scope in Python
Initialize all externally visible members of a Python class in the constructor
For Python interfaces that may be used outside a file, prefer docstrings over comments
Python comments should be reserved for code within a function, or interfaces that are local to a file
Use Google style docstrings for Python classes and functions, which can be parsed by Sphinx
Python attributes and variables can be documented inline with type and description (e.g.,self.x = 5followed by"""<type>: Description of 'x'""")
Avoid using reflection in Python when functionality can be easily achieved without reflection
When using try-except blocks in Python, limit the except clause to the smallest set of specific errors possible instead of catching all exceptions
When using try-except blocks in Python to handle multiple possible variable types (duck-typing), keep the body of the try as small as possible and use the else block to implement the logic
Files:
tensorrt_llm/_torch/models/modeling_utils.pytensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_deepgemm.pytests/unittest/_torch/modules/test_fused_moe.pytensorrt_llm/_torch/models/modeling_gpt_oss.pytensorrt_llm/_torch/model_config.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.pytensorrt_llm/_torch/modules/fused_moe/communication/deep_ep_low_latency.pytensorrt_llm/_torch/models/modeling_hunyuan_moe.pytensorrt_llm/_torch/models/modeling_deepseekv3.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_vanilla.pytensorrt_llm/_torch/modules/fused_moe/communication/base.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.pytensorrt_llm/_torch/modules/fused_moe/communication/deep_ep.pytensorrt_llm/_torch/modules/fused_moe/create_moe.pytensorrt_llm/_torch/modules/fused_moe/configurable_moe.pytensorrt_llm/_torch/modules/fused_moe/communication/__init__.pytensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.pytensorrt_llm/_torch/modules/fused_moe/communication/communication_factory.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.pytensorrt_llm/_torch/modules/fused_moe/interface.py
**/*.{cpp,h,cu,py}
📄 CodeRabbit inference engine (CODING_GUIDELINES.md)
All TensorRT-LLM Open Source Software code files should contain an NVIDIA copyright header that includes the current year at the top
Files:
tensorrt_llm/_torch/models/modeling_utils.pytensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_deepgemm.pytests/unittest/_torch/modules/test_fused_moe.pytensorrt_llm/_torch/models/modeling_gpt_oss.pytensorrt_llm/_torch/model_config.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.pytensorrt_llm/_torch/modules/fused_moe/communication/deep_ep_low_latency.pytensorrt_llm/_torch/models/modeling_hunyuan_moe.pytensorrt_llm/_torch/models/modeling_deepseekv3.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_vanilla.pytensorrt_llm/_torch/modules/fused_moe/communication/base.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.pytensorrt_llm/_torch/modules/fused_moe/communication/deep_ep.pytensorrt_llm/_torch/modules/fused_moe/create_moe.pytensorrt_llm/_torch/modules/fused_moe/configurable_moe.pytensorrt_llm/_torch/modules/fused_moe/communication/__init__.pytensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.pytensorrt_llm/_torch/modules/fused_moe/communication/communication_factory.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.pytensorrt_llm/_torch/modules/fused_moe/interface.py
🧠 Learnings (17)
📓 Common learnings
Learnt from: djns99
Repo: NVIDIA/TensorRT-LLM PR: 6915
File: cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu:4010-4012
Timestamp: 2025-08-14T23:23:27.449Z
Learning: For MOE (Mixture of Experts) code reviews in TensorRT-LLM, avoid repeatedly suggesting finalize fusion validation checks and safety assertions. The user djns99 has indicated these suggestions are repetitive and unwanted across multiple MOE-related changes.
📚 Learning: 2025-10-20T16:54:09.824Z
Learnt from: nvchenghaoz
Repo: NVIDIA/TensorRT-LLM PR: 8469
File: tensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.py:6-6
Timestamp: 2025-10-20T16:54:09.824Z
Learning: In tensorrt_llm/_torch/auto_deploy/custom_ops/rms_norm.py, the import `from ...modules.mamba.layernorm_gated import _layer_norm_fwd` is correct and should not be changed to modules.fla.layernorm_gated. The _layer_norm_fwd function exists in both modules/mamba/layernorm_gated.py and modules/fla/layernorm_gated.py, but the mamba version is the intended implementation for this use case.
Applied to files:
tensorrt_llm/_torch/models/modeling_utils.py
📚 Learning: 2025-08-21T02:39:12.009Z
Learnt from: djns99
Repo: NVIDIA/TensorRT-LLM PR: 7104
File: cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu:1475-1480
Timestamp: 2025-08-21T02:39:12.009Z
Learning: The min latency mode functionality in TensorRT-LLM MOE kernels (cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu) is deprecated and no longer being maintained/updated, as confirmed by djns99. Bug reports and optimization suggestions for the computeStridesTmaWarpSpecializedLowLatencyKernel and related min latency code paths should be deprioritized.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py
📚 Learning: 2025-09-02T13:42:44.885Z
Learnt from: pcastonguay
Repo: NVIDIA/TensorRT-LLM PR: 7455
File: tensorrt_llm/_torch/pyexecutor/py_executor.py:1852-1860
Timestamp: 2025-09-02T13:42:44.885Z
Learning: In MPI communication within TensorRT-LLM pipeline parallelism, different communication types (tokens, logits, termination sync) must use disjoint tag namespaces to avoid message routing collisions when using the same source/destination patterns.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.pytensorrt_llm/_torch/modules/fused_moe/communication/__init__.pytensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.py
📚 Learning: 2025-08-14T23:23:27.449Z
Learnt from: djns99
Repo: NVIDIA/TensorRT-LLM PR: 6915
File: cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu:4010-4012
Timestamp: 2025-08-14T23:23:27.449Z
Learning: For MOE (Mixture of Experts) code reviews in TensorRT-LLM, avoid repeatedly suggesting finalize fusion validation checks and safety assertions. The user djns99 has indicated these suggestions are repetitive and unwanted across multiple MOE-related changes.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/fused_moe_deepgemm.pytensorrt_llm/_torch/models/modeling_hunyuan_moe.pytensorrt_llm/_torch/models/modeling_deepseekv3.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_vanilla.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.pytensorrt_llm/_torch/modules/fused_moe/create_moe.pytensorrt_llm/_torch/modules/fused_moe/configurable_moe.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.pytensorrt_llm/_torch/modules/fused_moe/interface.py
📚 Learning: 2025-08-09T20:57:04.084Z
Learnt from: sklevtsov-nvidia
Repo: NVIDIA/TensorRT-LLM PR: 3294
File: cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_tma_warp_specialized_input.cu:118-127
Timestamp: 2025-08-09T20:57:04.084Z
Learning: In the CUTLASS MoE finalize fusion implementation (cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_tma_warp_specialized_input.cu), when setting `fused_finalize_epilogue.stride_final_output` with shape `(hidden_size, num_output_tokens, 1)`, the `num_rows_in_final_output` should be set to `num_output_tokens` (not `hidden_size`) because of a swap+transpose operation that maps rows of the output tensor to `hidden_size` and columns to `num_output_tokens`.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/fused_moe_deepgemm.pytensorrt_llm/_torch/model_config.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_vanilla.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.pytensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.py
📚 Learning: 2025-08-19T03:35:20.866Z
Learnt from: djns99
Repo: NVIDIA/TensorRT-LLM PR: 6915
File: cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu:4616-4626
Timestamp: 2025-08-19T03:35:20.866Z
Learning: In the MOE profiler TMA workspace preparation (cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu), the overlapping of TMA WS regions for NONE and FINALIZE variants is deliberate design to save memory space, as confirmed by djns99. The comment "reuse the same pointers to save space" reflects this intentional behavior.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/fused_moe_deepgemm.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
📚 Learning: 2025-08-08T22:03:40.707Z
Learnt from: sklevtsov-nvidia
Repo: NVIDIA/TensorRT-LLM PR: 3294
File: cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu:1198-1209
Timestamp: 2025-08-08T22:03:40.707Z
Learning: In the CUTLASS MoE kernels (cpp/tensorrt_llm/cutlass_extensions), when `layout_info.fusion` is set to `TmaWarpSpecializedGroupedGemmInput::EpilogueFusion::FINALIZE`, the `router_scales` parameter must be non-null by design. The fused finalize kernel epilogue does not perform nullptr checks and requires valid router scales to function correctly. This is an implicit contract that callers must satisfy when enabling the FINALIZE fusion mode.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/fused_moe_deepgemm.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py
📚 Learning: 2025-08-26T06:07:02.166Z
Learnt from: shaharmor98
Repo: NVIDIA/TensorRT-LLM PR: 7231
File: tensorrt_llm/_torch/pyexecutor/_util.py:504-509
Timestamp: 2025-08-26T06:07:02.166Z
Learning: In tensorrt_llm/_torch/pyexecutor/_util.py, when calling model_engine.set_lora_model_config(), pass model_binding_config.mlp_hidden_size directly without multiplying by mapping.tp_size, as the mlp_hidden_size from get_bindings_model_config() is already the per-TP rank value needed for LoRA weight packaging.
Applied to files:
tensorrt_llm/_torch/model_config.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
📚 Learning: 2025-08-21T21:48:35.135Z
Learnt from: djns99
Repo: NVIDIA/TensorRT-LLM PR: 7104
File: cpp/tensorrt_llm/cutlass_extensions/include/cutlass_extensions/epilogue/fusion/sm90_visitor_scatter.hpp:399-417
Timestamp: 2025-08-21T21:48:35.135Z
Learning: CUTLASS extensions in TensorRT-LLM (located under cpp/tensorrt_llm/cutlass_extensions/) are designed to integrate with and extend functionality in the external CUTLASS repository. When analyzing these extensions, their consumers and functionality wiring may exist in the CUTLASS codebase rather than within TensorRT-LLM itself.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.py
📚 Learning: 2025-11-14T11:22:03.729Z
Learnt from: nzmora-nvidia
Repo: NVIDIA/TensorRT-LLM PR: 9163
File: tensorrt_llm/_torch/auto_deploy/custom_ops/quant.py:107-113
Timestamp: 2025-11-14T11:22:03.729Z
Learning: In TensorRT-LLM AutoDeploy custom ops, when adding hardware capability checks to select between kernel implementations (e.g., cuBLAS vs. CUDA kernel), use descriptive variable names that identify the specific GPU architectures or families being targeted (e.g., `is_blackwell_geforce_or_ada`) rather than generic names like `enable_cuda_core`. This makes it clear that the code is selecting an implementation path based on hardware capabilities, not enabling/disabling hardware features.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/communication/deep_ep_low_latency.py
📚 Learning: 2025-08-14T06:36:40.701Z
Learnt from: timlee0212
Repo: NVIDIA/TensorRT-LLM PR: 6886
File: tensorrt_llm/_torch/models/modeling_deepseekv3.py:0-0
Timestamp: 2025-08-14T06:36:40.701Z
Learning: In DeepSeek V3 model (tensorrt_llm/_torch/models/modeling_deepseekv3.py), the disagreement between AllReduce.__init__ guard and _compute_mlp_tp_size logic for MNNVL usage is expected by design. The AllReduce component and MLP TP-size computation intentionally use different criteria for MNNVL availability decisions.
Applied to files:
tensorrt_llm/_torch/models/modeling_deepseekv3.py
📚 Learning: 2025-09-24T03:31:28.908Z
Learnt from: tongyuantongyu
Repo: NVIDIA/TensorRT-LLM PR: 7520
File: tensorrt_llm/_torch/pyexecutor/resource_manager.py:605-613
Timestamp: 2025-09-24T03:31:28.908Z
Learning: In TensorRT-LLM Ray orchestrator mode, ProcessGroups are initialized with both Gloo and NCCL backends (e.g., "cuda:nccl,cpu:gloo"), allowing PyTorch distributed to automatically route CPU tensors through Gloo and GPU tensors through NCCL. This eliminates the need for manual device placement when performing allreduce operations on base types.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/communication/__init__.py
📚 Learning: 2025-09-22T19:25:45.607Z
Learnt from: nv-lschneider
Repo: NVIDIA/TensorRT-LLM PR: 7910
File: cpp/tensorrt_llm/kernels/userbuffers/ub_allocator.cpp:170-179
Timestamp: 2025-09-22T19:25:45.607Z
Learning: In NCCLUserBufferAllocator::getNCCLDevComm(), multimem support is hard-coded to true because multimem is required for this function. The caller is responsible for ensuring multimem is available before calling this function - it should not be called if multimem is not supported.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/communication/__init__.py
📚 Learning: 2025-09-23T14:58:05.372Z
Learnt from: nv-lschneider
Repo: NVIDIA/TensorRT-LLM PR: 7910
File: cpp/tensorrt_llm/kernels/nccl_device/config.cu:42-49
Timestamp: 2025-09-23T14:58:05.372Z
Learning: In TensorRT-LLM NCCL device kernels (cpp/tensorrt_llm/kernels/nccl_device/), the token partitioning intentionally uses ceil-like distribution (same token_per_rank for all ranks) to ensure all ranks launch the same number of blocks. This is required for optimal NCCL device API barrier performance, even though it may launch extra blocks for non-existent tokens on later ranks. Runtime bounds checking in the kernel (blockID validation) handles the overshoot cases.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.py
📚 Learning: 2025-09-19T21:28:13.751Z
Learnt from: jhaotingc
Repo: NVIDIA/TensorRT-LLM PR: 7856
File: cpp/tensorrt_llm/thop/fp8BlockScaleMoe.cpp:159-166
Timestamp: 2025-09-19T21:28:13.751Z
Learning: In TensorRT-LLM blockScaleMoe routing (cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/runner.cu), the DeepSeek routing method performs reinterpret_cast<float*>(routingLogits) at line 89, which could cause issues if routing_logits are BF16. However, Qwen3-FP8 models use RenormalizeNaive routing method and are not affected by this dtype casting issue.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py
📚 Learning: 2025-10-20T17:07:18.745Z
Learnt from: nvchenghaoz
Repo: NVIDIA/TensorRT-LLM PR: 8469
File: tensorrt_llm/_torch/auto_deploy/models/patches/nemotron_h.py:98-116
Timestamp: 2025-10-20T17:07:18.745Z
Learning: In NemotronH models (tensorrt_llm/_torch/auto_deploy/models/patches/nemotron_h.py), the gate (self.gate) returns topk_indices and topk_weights that are already in the correct shape to be passed directly to torch_ops.auto_deploy.torch_moe without needing to reshape them when hidden_states is flattened.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py
🧬 Code graph analysis (12)
tensorrt_llm/_torch/models/modeling_utils.py (1)
tensorrt_llm/_torch/modules/fused_moe/interface.py (1)
MoE(113-744)
tensorrt_llm/_torch/models/modeling_gpt_oss.py (1)
tensorrt_llm/_torch/modules/fused_moe/interface.py (1)
MoE(113-744)
tensorrt_llm/_torch/model_config.py (2)
tests/unittest/_torch/modeling/test_modeling_out_of_tree.py (1)
max_num_tokens(63-66)tensorrt_llm/mapping.py (1)
dp_size(226-227)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.py (4)
tensorrt_llm/mapping.py (1)
dp_size(226-227)tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py (1)
quantize_input(275-321)tensorrt_llm/_torch/modules/fused_moe/interface.py (7)
quantize_input(507-535)has_w4a8_mxfp4_fp8(702-705)_(84-110)has_deepseek_fp8_block_scales(684-687)has_w4a16_mxfp4(714-717)has_nvfp4(690-693)has_w4a8_mxfp4_mxfp8(708-711)tensorrt_llm/_utils.py (2)
shape(989-990)shape(1006-1007)
tensorrt_llm/_torch/modules/fused_moe/communication/deep_ep_low_latency.py (3)
tensorrt_llm/_torch/modules/fused_moe/communication/deep_ep.py (1)
is_platform_supported(73-79)tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.py (1)
is_platform_supported(172-176)tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.py (1)
is_platform_supported(75-79)
tensorrt_llm/_torch/models/modeling_hunyuan_moe.py (2)
tensorrt_llm/_torch/modules/fused_moe/interface.py (1)
MoE(113-744)tensorrt_llm/_torch/modules/fused_moe/create_moe.py (1)
create_moe(266-387)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py (2)
tests/unittest/_torch/modeling/test_modeling_out_of_tree.py (1)
max_num_tokens(63-66)tensorrt_llm/mapping.py (1)
dp_size(226-227)
tensorrt_llm/_torch/modules/fused_moe/communication/deep_ep.py (3)
tensorrt_llm/_torch/modules/fused_moe/communication/deep_ep_low_latency.py (1)
is_platform_supported(86-94)tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.py (1)
is_platform_supported(172-176)tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.py (1)
is_platform_supported(75-79)
tensorrt_llm/_torch/modules/fused_moe/create_moe.py (4)
tensorrt_llm/_torch/modules/fused_moe/configurable_moe.py (1)
ConfigurableMoE(53-1089)tensorrt_llm/_torch/modules/fused_moe/interface.py (2)
MoE(113-744)MoEWeightLoadingMode(16-22)tensorrt_llm/_torch/model_config.py (2)
torch_dtype(174-179)ModelConfig(75-621)tensorrt_llm/logger.py (1)
warning(132-133)
tensorrt_llm/_torch/modules/fused_moe/communication/__init__.py (2)
tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.py (1)
NVLinkOneSided(37-396)tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.py (1)
NVLinkTwoSided(35-201)
tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.py (4)
tensorrt_llm/_torch/modules/fused_moe/communication/base.py (3)
Communication(34-163)supports_post_quant_dispatch(68-78)is_workload_feasible(55-66)tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.py (3)
is_platform_supported(75-79)supports_post_quant_dispatch(81-85)is_workload_feasible(87-94)tensorrt_llm/_mnnvl_utils.py (2)
MnnvlMemory(53-338)supports_mnnvl(332-338)tensorrt_llm/llmapi/llm_args.py (2)
ep_rank(311-315)ep_size(318-322)
tensorrt_llm/_torch/modules/fused_moe/communication/communication_factory.py (5)
tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.py (2)
NVLinkOneSided(37-396)is_platform_supported(172-176)tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.py (2)
NVLinkTwoSided(35-201)is_platform_supported(75-79)tensorrt_llm/_torch/modules/fused_moe/communication/base.py (1)
Communication(34-163)tensorrt_llm/_torch/modules/fused_moe/communication/deep_ep.py (2)
is_platform_supported(73-79)DeepEP(35-241)tensorrt_llm/_torch/modules/fused_moe/communication/deep_ep_low_latency.py (1)
is_platform_supported(86-94)
🪛 Ruff (0.14.5)
tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.py
87-87: Unused method argument: all_rank_num_tokens
(ARG002)
87-87: Unused method argument: num_chunks
(ARG002)
144-146: Avoid specifying long messages outside the exception class
(TRY003)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.py
260-260: Unused method argument: kwargs
(ARG002)
311-313: Avoid specifying long messages outside the exception class
(TRY003)
tensorrt_llm/_torch/modules/fused_moe/create_moe.py
273-273: Do not perform function call ModelConfig in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
365-367: Avoid specifying long messages outside the exception class
(TRY003)
tensorrt_llm/_torch/modules/fused_moe/configurable_moe.py
101-101: Do not perform function call ModelConfig in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
234-234: Unused method argument: model_config
(ARG002)
379-379: Unused method argument: kwargs
(ARG002)
724-724: zip() without an explicit strict= parameter
Add explicit value for parameter strict=
(B905)
814-814: Avoid specifying long messages outside the exception class
(TRY003)
818-822: Avoid specifying long messages outside the exception class
(TRY003)
tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.py
184-184: Unused method argument: all_rank_num_tokens
(ARG002)
184-184: Unused method argument: num_chunks
(ARG002)
tensorrt_llm/_torch/modules/fused_moe/communication/communication_factory.py
145-148: Avoid specifying long messages outside the exception class
(TRY003)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py
561-561: Local variable mxfp8_x is assigned to but never used
Remove assignment to unused variable mxfp8_x
(F841)
561-561: Local variable sf is assigned to but never used
Remove assignment to unused variable sf
(F841)
611-611: Unused method argument: kwargs
(ARG002)
757-759: Avoid specifying long messages outside the exception class
(TRY003)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
Collaborator
Author
|
/bot run |
nekorobov
reviewed
Nov 26, 2025
dongxuy04
reviewed
Nov 26, 2025
xxi-nv
changed the title
Collaborator
Author
|
/bot run |
1 similar comment
Member
|
/bot run |
xxi-nv
force-pushed
the
refactorTRTLLMGen
branch
from
b93abcd to
7e09d1b
Compare
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #25919 [ run ] triggered by Bot. Commit: |
xxi-nv
force-pushed
the
refactorTRTLLMGen
branch
from
7e09d1b to
97d1428
Compare
Collaborator
|
PR_Github #25919 [ run ] completed with state |
xxi-nv
force-pushed
the
refactorTRTLLMGen
branch
from
97d1428 to
24b05e8
Compare
bobboli
reviewed
Nov 27, 2025
tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.py
Outdated
Show resolved
Hide resolved
tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.py
Outdated
Show resolved
Hide resolved
bobboli
reviewed
Nov 27, 2025
tensorrt_llm/_torch/modules/fused_moe/communication/deep_ep_low_latency.py
Show resolved
Hide resolved
tensorrt_llm/_torch/modules/fused_moe/communication/communication_factory.py
Outdated
Show resolved
Hide resolved
tensorrt_llm/_torch/modules/fused_moe/communication/communication_factory.py
Outdated
Show resolved
Hide resolved
tensorrt_llm/_torch/modules/fused_moe/communication/communication_factory.py
Outdated
Show resolved
Hide resolved
Collaborator
Author
|
/bot run |
1 similar comment
Member
|
/bot run |
xxi-nv
force-pushed
the
refactorTRTLLMGen
branch
from
d4f64d5 to
ba75b99
Compare
Collaborator
|
PR_Github #26114 [ run ] triggered by Bot. Commit: |
dongxuy04
approved these changes
Nov 28, 2025
bobboli
approved these changes
Nov 28, 2025
tomeras91
approved these changes
Nov 28, 2025
Collaborator
|
PR_Github #26114 [ run ] completed with state |
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26175 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #26175 [ run ] completed with state |
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26195 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #26195 [ run ] completed with state |
…he TRTLLMGenFusedMoE as the backend in ConfigurableMoE Signed-off-by: xxi <[email protected]> modified: tensorrt_llm/_torch/model_config.py modified: tensorrt_llm/_torch/models/modeling_deepseekv3.py modified: tensorrt_llm/_torch/models/modeling_gpt_oss.py modified: tensorrt_llm/_torch/models/modeling_hunyuan_moe.py modified: tensorrt_llm/_torch/models/modeling_utils.py modified: tensorrt_llm/_torch/modules/fused_moe/communication/__init__.py modified: tensorrt_llm/_torch/modules/fused_moe/communication/allgather_reducescatter.py modified: tensorrt_llm/_torch/modules/fused_moe/communication/base.py modified: tensorrt_llm/_torch/modules/fused_moe/communication/communication_factory.py modified: tensorrt_llm/_torch/modules/fused_moe/communication/deep_ep.py modified: tensorrt_llm/_torch/modules/fused_moe/communication/deep_ep_low_latency.py renamed: tensorrt_llm/_torch/modules/fused_moe/communication/mnnvl_throughput.py -> tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_one_sided.py renamed: tensorrt_llm/_torch/modules/fused_moe/communication/mnnvl_latency.py -> tensorrt_llm/_torch/modules/fused_moe/communication/nvlink_two_sided.py new file: tensorrt_llm/_torch/modules/fused_moe/configurable_moe.py modified: tensorrt_llm/_torch/modules/fused_moe/create_moe.py modified: tensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.py modified: tensorrt_llm/_torch/modules/fused_moe/fused_moe_deepgemm.py modified: tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py modified: tensorrt_llm/_torch/modules/fused_moe/fused_moe_vanilla.py modified: tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py modified: tensorrt_llm/_torch/modules/fused_moe/interface.py modified: tests/unittest/_torch/modeling/test_modeling_nemotron_h.py modified: tests/unittest/_torch/modules/test_fused_moe.py
xxi-nv
force-pushed
the
refactorTRTLLMGen
branch
from
cc08b4a to
17666b9
Compare
Collaborator
Author
Yeah, actually, I will add several tests to make sure we can at least cover the ConfigurableMoE functional works in the following PR. |
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26231 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #26231 [ run ] completed with state |
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26257 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #26257 [ run ] completed with state |
Superjomn
pushed a commit
to hchings/TensorRT-LLM
that referenced
this pull request
Dec 1, 2025
…port TRTLLMGenFusedMoE as backend (NVIDIA#9486)
1 task
MinaHuai
pushed a commit
to davidmlw/TensorRT-LLM
that referenced
this pull request
Dec 10, 2025
…VIDIA#8779) The performance results of some kernels could be easily affected by the warm/cold L2 cache status. To achieve more precise profiling results, the L2 cache is cleared for every execution by the circular buffer method for better benchmarking during autotuning. Signed-off-by: Yukun He <[email protected]> [None][infra] Waive failed cases for main branch on 11/25 (NVIDIA#9429) Signed-off-by: qqiao <[email protected]> [NVIDIA#8391][chore] test_perf.py to lock clocks read from gpu_configs.yml instead of max freq (NVIDIA#9409) Signed-off-by: Eran Geva <[email protected]> [None][ci] Move more test stages to use OCI machines (NVIDIA#9395) Signed-off-by: Yanchao Lu <[email protected]> Co-authored-by: Matt Lefebvre <[email protected]> [None][feat] Improve TRTLLM MoE in small hidden size throughput cases (NVIDIA#9377) Signed-off-by: Anthony Chang <[email protected]> [https://nvbugs/5537996][fix] Let KV cache manager block initialization be aware whether it is doing a dry run or not (NVIDIA#9093) Before this commit, the kv cache manager does the same regardless, which causes a mis-calculation in free memory available to allocate for the KV cache manager, hence causing a crash. This commit fixes this by letting KV cache manager initialization be aware whether it is doing the dry run or not. If it is a dry run, use the max_tokens setting that is already pre-calculated and filled into kv_cache_config.max_tokens. Signed-off-by: eopXD <[email protected]> [https://nvbugs/5667922][fix] Update long context evaluation config (NVIDIA#9426) Signed-off-by: mni <[email protected]> [None][fix] Mitigate test timeout issues (NVIDIA#9445) Signed-off-by: Shixiaowei02 <[email protected]> [None][chore] Fix trtllm-eval for PyTorchLLM (NVIDIA#9427) Signed-off-by: Fanrong Li <[email protected]> [None][feat] Add a parser to layer-wise benchmarks (NVIDIA#9440) Signed-off-by: Tailing Yuan <[email protected]> [None][feat] Support custom chat template for tool calling (NVIDIA#9297) Signed-off-by: Pengyun Lin <[email protected]> [TRTLLM-8160][feat] Add draft token tree runtime on CDL (NVIDIA#8586) Signed-off-by: Yue Weng <[email protected]> [None][ci] waive a test (NVIDIA#9458) Signed-off-by: Yan Chunwei <[email protected]> [https://nvbugs/5680905][fix] Relax the MMLU accuracy requirement for DS-v3.2 (NVIDIA#9439) Signed-off-by: Fanrong Li <[email protected]> [TRTLLM-8376][feat] top-p optimization (removes redundant softmax) (NVIDIA#9411) Signed-off-by: ixlmar <[email protected]> [TRTLLM-9490][feat] use FlashInfer's top_k_sampling_from_probs (NVIDIA#9457) Signed-off-by: ixlmar <[email protected]> [https://nvbugs/5647400] [fix] Enlarged the AllReduce workspace size to 64MB. Added AllReduce strategy to AD config. (NVIDIA#9145) Signed-off-by: Eran Geva <[email protected]> [TRTLLM-909][feat] Overlap context chunks in pipeline parallel mode (NVIDIA#9308) Signed-off-by: Robin Kobus <[email protected]> [None][chore] AutoDeploy add multi stream moe pass to default.yaml (NVIDIA#9430) Signed-off-by: Suyog Gupta <[email protected]> [https://nvbugs/5685143][fix] avoid cudaFree overlap with cuda graph (NVIDIA#9438) Signed-off-by: Chuang Zhu <[email protected]> [None][chore] Bump version to 1.2.0rc5 (NVIDIA#9455) Signed-off-by: Yiqing Yan <[email protected]> [TRTLLM-8936][test] Add disagg and wideep multi-node multi-gpu test cases (NVIDIA#9356) Signed-off-by: FredricZ-2007 <[email protected]> [None][ci] move some slow test cases of DGX-B200 to post merge (NVIDIA#9467) Signed-off-by: junq <[email protected]> [TRTLLM-9293][feat] Enable partial weight loading to support streaming update weights (NVIDIA#9224) Signed-off-by: shuyix <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [TRTLLM-9264][fix] Add accuracy/unit tests/doc for phi4mm (NVIDIA#9246) Signed-off-by: Wanli Jiang <[email protected]> [https://nvbugs/5580099][fix] Cherry pick IMA issue fix from release/1.1 (NVIDIA#9032) Signed-off-by: Junyi Xu <[email protected]> [None][chore] Upgrade CuteDSL to 4.3.0 (NVIDIA#9444) Signed-off-by: Enwei Zhu <[email protected]> [None][feat] Support MLA chunked prefill for DeepSeek V3.2 model (NVIDIA#9376) Signed-off-by: Chang Liu (Enterprise Products) <[email protected]> [None][feat] Add environment variable to force spec-dec number of accepted tokens (NVIDIA#9371) Signed-off-by: Aurelien Chartier <[email protected]> [None][infra] Update allowed list 2025.11.25 (NVIDIA#9468) Signed-off-by: Yuanjing Xue <[email protected]> [None][infra] Fail the pipeline when slurm ssh dropped (NVIDIA#9157) Signed-off-by: Yuanjing Xue <[email protected]> [None][feat] AutoDeploy: Remove redundant copies in mamba layers (NVIDIA#9461) Signed-off-by: Chenghao Zhang <[email protected]> Co-authored-by: Suyog Gupta <[email protected]> [None][feat] AutoDeploy: Add A_log fusion for Mamba layers (NVIDIA#9422) Signed-off-by: Chenghao Zhang <[email protected]> [None][ci] Waive blackwell test on spec gate. (NVIDIA#9502) Signed-off-by: Zheyu Fu <[email protected]> [https://nvbugs/5608930][fix] Fix a typo (NVIDIA#9487) Signed-off-by: Shixiaowei02 <[email protected]> [NVIDIA#9463][feat] Add revision option to trtllm commands (NVIDIA#9498) Signed-off-by: Aurelien Chartier <[email protected]> [TRTLLM-9085][doc] fix math formula rendering issues (NVIDIA#9481) Signed-off-by: junq <[email protected]> [None][chore] update comments in llm_args.py (NVIDIA#9472) Signed-off-by: junq <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [https://nvbugs/5680310][fix] Fix ctx only timed out test (NVIDIA#9410) Signed-off-by: Patrice Castonguay <[email protected]> [https://nvbugs/5547414][fix] enable case after using local cache model (NVIDIA#9473) Signed-off-by: Hui Gao <[email protected]> [None][fix] Replace PYTORCH_CUDA_ALLOC_CONF with PYTORCH_ALLOC_CONF to fix deprecation warning (NVIDIA#9294) Signed-off-by: Jiagan Cheng <[email protected]> [https://nvbugs/5698581][fix] Init draft tokens for CUDA graph dummy request (NVIDIA#9505) Signed-off-by: ziyixiong-nv <[email protected]> [None][infra] Waive failed case in pre-merge on 11/27 (NVIDIA#9507) Signed-off-by: qqiao <[email protected]> [TRTLLM-9513][docs] Qwen3 deployment guide (NVIDIA#9488) Signed-off-by: Lanyu Liao <[email protected]> Co-authored-by: Lanyu Liao <[email protected]> [None][chore] revert batch_size=1 to prevent timeout and lower accuracy reference by 0.12% as a WAR (NVIDIA#9447) Signed-off-by: Lizhi Zhou <[email protected]> Co-authored-by: Shi Xiaowei <[email protected]> [TRTLLM-9279][infra] Use flexcache for gh200 nodes since they locate in Austin (NVIDIA#9405) Signed-off-by: qqiao <[email protected]> Signed-off-by: Emma Qiao <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> [cherry-pick][https://nvbugs/5670793][fix] Solve trtllm-serve launch_disaggregated issue (NVIDIA#9346) Signed-off-by: xxi <[email protected]> [None][infra] Fix Slurm job script (NVIDIA#9508) Signed-off-by: Yuanjing Xue <[email protected]> [None][fix] change allreduce workspace dtype to torch.int64 to avoid overflow (NVIDIA#9479) Signed-off-by: Zhenhuan Chen <[email protected]> [None][feat] add qwen3-next CI test of accuracy on BF16 and NVFP4 (NVIDIA#9330) Signed-off-by: jiant <[email protected]> [None][fix] fix TP support for DeepSeek-V3.2 on hopper (NVIDIA#9484) Signed-off-by: Fanrong Li <[email protected]> [TRTLLM-9389][chore] Refactor AlltoallMethodType. (NVIDIA#9388) Signed-off-by: Bo Li <[email protected]> [https://nvbugs/5674665][chore] Add test coverage for https://nvbugspro.nvidia.com/bug/5674665 (NVIDIA#9518) Signed-off-by: eopXD <[email protected]> [TRTLLM-7288][infra] Download merged waive list in slurm script (NVIDIA#8999) Signed-off-by: Yiqing Yan <[email protected]> Signed-off-by: Yanchao Lu <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> [https://nvbugs/5687820][fix] Remove self.abort() in DetokenizedGenerationResult (NVIDIA#9449) Signed-off-by: Enwei Zhu <[email protected]> [NVIDIA#9150][feat] AutoDeploy Nemotron-Flash support (NVIDIA#9504) Signed-off-by: Lucas Liebenwein <[email protected]> [None] [chore] Update to cutlass 4.3 (NVIDIA#8637) Signed-off-by: Kaiyu Xie <[email protected]> [https://nvbugs/5637037][chore] Update waive lists. (NVIDIA#9386) Signed-off-by: Bo Li <[email protected]> Signed-off-by: Enwei Zhu <[email protected]> Co-authored-by: Enwei Zhu <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [TRTLLM-8970][infra] Fix generate report when has isolation test result (NVIDIA#8861) Signed-off-by: qqiao <[email protected]> Signed-off-by: Emma Qiao <[email protected]> [https://nvbugs/5685015][fix] Update invalid max_token test (NVIDIA#9435) Signed-off-by: Junyi Xu <[email protected]> [None][fix] Fix on-disk cache and revise logger/statistics for AutoTuner. (NVIDIA#9211) Signed-off-by: Yukun He <[email protected]> [https://nvbugs/5689658][test] Fix gpu lock issue running on cluster (NVIDIA#9441) Signed-off-by: yufeiwu <[email protected]> [None][chore] add spec_decoding configs in perf benchmark scripts and fix typos (NVIDIA#9533) Signed-off-by: Lanyu Liao <[email protected]> Co-authored-by: Lanyu Liao <[email protected]> [None][fix] Remove FP8 K/V buffer from TRTLLM sparse MLA attention kernel (NVIDIA#9529) Signed-off-by: Chang Liu (Enterprise Products) <[email protected]> [None] [chore] Enhancements and clean up to slurm scripts (NVIDIA#9493) Signed-off-by: Kaiyu Xie <[email protected]> [None][chore] Revert "[None][fix] change allreduce workspace dtype to torch.int64 t… (NVIDIA#9538) Signed-off-by: Zhenhuan Chen <[email protected]> [None][infra] Waive failed cases for main branch on 11/28 (NVIDIA#9539) Signed-off-by: qqiao <[email protected]> [None][fix] Pass checkpoint_format to create_input_processor (NVIDIA#9521) Signed-off-by: Robin Kobus <[email protected]> [TRTLLM-9541][infra] Use artifactory mirror for download.pytorch.org (NVIDIA#9477) Signed-off-by: ZhanruiSunCh <[email protected]> Signed-off-by: Zhanrui Sun <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> [TRTLLM-9488][feat] add 'disable_flashinfer_sampling' config option (NVIDIA#9454) Signed-off-by: ixlmar <[email protected]> [None][infra] Waive failed case in pre-merge on 11/28 (NVIDIA#9537) Signed-off-by: Wangshanshan <[email protected]> [None][perf] Helix: improve all-to-all perf for large CP size (NVIDIA#9494) Signed-off-by: Matthias Jouanneaux <[email protected]> Signed-off-by: Zheyu Fu <[email protected]> Co-authored-by: Zheyu Fu <[email protected]> [None][feat] support for more accurate AR calculation (NVIDIA#9323) Signed-off-by: binghanc <[email protected]> [TRTLLM-9488][fix] llmapi references (NVIDIA#9547) Signed-off-by: ixlmar <[email protected]> [NVIDIA#8948][feat] Support custom sharding config (NVIDIA#9143) Signed-off-by: greg-kwasniewski1 <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [None][chore] Weekly mass integration of release/1.1 -- rebase (NVIDIA#9522) Signed-off-by: yunruis <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Wangshanshan <[email protected]> Signed-off-by: qgai <[email protected]> Signed-off-by: Balaram Buddharaju <[email protected]> Signed-off-by: Yan Chunwei <[email protected]> Signed-off-by: Junyi Xu <[email protected]> Signed-off-by: Simeng Liu <[email protected]> Signed-off-by: nv-guomingz <[email protected]> Signed-off-by: Jin Li <[email protected]> Signed-off-by: Ivy Zhang <[email protected]> Signed-off-by: Vincent Zhang <[email protected]> Signed-off-by: peaceh <[email protected]> Signed-off-by: Michal Guzek <[email protected]> Signed-off-by: Michal Guzek <[email protected]> Signed-off-by: Chang Liu (Enterprise Products) <[email protected]> Signed-off-by: leslie-fang25 <[email protected]> Signed-off-by: Shunkang <[email protected]> Signed-off-by: junq <[email protected]> Co-authored-by: yunruis <[email protected]> Co-authored-by: sunnyqgg <[email protected]> Co-authored-by: brb-nv <[email protected]> Co-authored-by: Yan Chunwei <[email protected]> Co-authored-by: JunyiXu-nv <[email protected]> Co-authored-by: Simeng Liu <[email protected]> Co-authored-by: Guoming Zhang <[email protected]> Co-authored-by: Jin Li <[email protected]> Co-authored-by: Ivy Zhang <[email protected]> Co-authored-by: Vincent Zhang <[email protected]> Co-authored-by: peaceh-nv <[email protected]> Co-authored-by: Michal Guzek <[email protected]> Co-authored-by: Chang Liu <[email protected]> Co-authored-by: Leslie Fang <[email protected]> Co-authored-by: Shunkangz <[email protected]> Co-authored-by: Shunkang <[email protected]> Co-authored-by: QI JUN <[email protected]> [TRTLLM-5971][feat] Integrate helix parallelism (NVIDIA#9342) Signed-off-by: Balaram Buddharaju <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [None][infra] - Request idle time exemption for OCI jobs (NVIDIA#9528) Signed-off-by: Yanchao Lu <[email protected]> [None][infra] Wiave failed tests for main branch on 11/30 (NVIDIA#9555) Signed-off-by: qqiao <[email protected]> [None][fix] Fix port conflict in disagg tests (NVIDIA#9474) Signed-off-by: Junyi Xu <[email protected]> [None][ci] Split H100_PCIe-PyTorch-Post-Merge test stage (NVIDIA#9558) Signed-off-by: Yanchao Lu <[email protected]> [None][ci] Split H100_PCIe-PyTorch-Post-Merge test stage (NVIDIA#9559) Signed-off-by: Yanchao Lu <[email protected]> [TRTLLM-8958][feat] and [TRTLLM-8960]: create ConfigurableMoE and support TRTLLMGenFusedMoE as backend (NVIDIA#9486) [None] [feat] Optimize the algorithm part of RocketKV (NVIDIA#9333) Signed-off-by: yuhangh <[email protected]> [https://nvbugs/5690172][fix] Fix Qwen3-235B ATP accuracy issue with PDL (NVIDIA#9530) Signed-off-by: Enwei Zhu <[email protected]> [TRTLLM-6222][feat] Extend cute_dsl_nvfp4_gemm to sm103. (NVIDIA#9543) Signed-off-by: Mindy Li <[email protected]> [None][fix] Correct virtual memory allocation alignment (NVIDIA#9491) Signed-off-by: Yuan Tong <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [https://nvbugs/5684703][fix] Unwaive disagg guided decoding test (NVIDIA#9466) Signed-off-by: Enwei Zhu <[email protected]> [https://nvbugs/5503479][fix] Temporarily lower reference accuracy to stabilize CI (NVIDIA#9398) Signed-off-by: Pengbo Wang <[email protected]> [None][chore] remove qwen3-next accuracy tests (NVIDIA#9534) Signed-off-by: jiant <[email protected]> [None][doc] fix mtp.py typo (NVIDIA#9307) Signed-off-by: liugaoji <[email protected]> [None][feat] add chat template kwargs support to longbench-v2 (NVIDIA#9544) Signed-off-by: Fanrong Li <[email protected]> [NVIDIA#9496][fix] AutoDeploy: remove auto-tuner from nvfp4_gemm forward (NVIDIA#9497) Signed-off-by: Neta Zmora <[email protected]> [None][fix] Replace hash method with unique_id for cutedsl MoE runners. (NVIDIA#9569) Signed-off-by: Yukun He <[email protected]> [None][chore] refactor disaggregated scripts to use named arguments (NVIDIA#9581) Signed-off-by: Zhenhuan Chen <[email protected]> [TRTLLM-6222][feat] Several perf opt for cuteDSL nvf4 gemm (NVIDIA#9428) Signed-off-by: Yuhan Li <[email protected]> [None][chore] reduce the layers of the `devel` docker image (NVIDIA#9077) Signed-off-by: Martin Marciniszyn Mehringer <[email protected]> [https://nvbugs/5651854][infra] Enable perf metrics during accuracy testing (NVIDIA#9140) [None][fix] Skip Allreduce init for Attention DP (NVIDIA#9542) Signed-off-by: Enwei Zhu <[email protected]> [None][test] [None][test] Waive main branch test failures 12/1 (NVIDIA#9566) Signed-off-by: Yanchao Lu <[email protected]> [None][ci] Minor change for Slurm scripts (NVIDIA#9561) Signed-off-by: Yanchao Lu <[email protected]> [TRTLLM-6768][infra] Fix params for not updating github status (NVIDIA#6747) Signed-off-by: Yiqing Yan <[email protected]> [None][infra] Update the pytest options after MI (NVIDIA#9579) Signed-off-by: qqiao <[email protected]> [TRTLLM-6756][feat] Add Beam Search to TorchSampler (NVIDIA#8509) Signed-off-by: Stefan Niebler <[email protected]> [None][chore] Defer exposing context parallel configs (NVIDIA#9552) Signed-off-by: Balaram Buddharaju <[email protected]> [TRTC-1943][feat] Env vars override support in LLM API (NVIDIA#9104) Signed-off-by: Venky Ganesh <[email protected]> [None][feat] AutoDeploy: Use the router gemm op for nemotron MOE (NVIDIA#9500) Signed-off-by: Chenghao Zhang <[email protected]> [NVIDIA#9198][feat] Refactor dist ops in AutoDeploy (NVIDIA#9301) Signed-off-by: Eran Geva <[email protected]> [None][fix] Prevent YAML partial kv_cache_config from incorrectly overriding the complete kv_cache_config (NVIDIA#9262) Signed-off-by: Yuening Li <[email protected]> [TRTLLM-9085][doc] fix math formula rendering issues in github (NVIDIA#9605) Signed-off-by: junq <[email protected]> [None][feat] Unify nvfp4 gemm backend (NVIDIA#8963) Signed-off-by: Shijie Wang <[email protected]> Signed-off-by: Yukun He <[email protected]> Signed-off-by: Shijie <[email protected]> Co-authored-by: Yukun He <[email protected]> [None][feat] Add support for KVCache reuse for DSv32 (NVIDIA#9383) Signed-off-by: Iman Tabrizian <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [None][chroe] Polish qwen3-next modeling code. (NVIDIA#8902) Signed-off-by: nv-guomingz <[email protected]> [https://nvbugs/5703953][fix] Use random port for disagg tests (NVIDIA#9582) Signed-off-by: Junyi Xu <[email protected]> [None][fix] Waive gb200 (NVIDIA#9580) Signed-off-by: Xin He (SW-GPU) <[email protected]> [FMDL-1328][feat] Add support for nano-v3 and super-v3 with pytorch backend (NVIDIA#9261) Signed-off-by: Wanli Jiang <[email protected]> [https://nvbugs/5582091][test] increase warmup times in testing for multi-gpu cases (NVIDIA#9578) Signed-off-by: Ruodi Lu <[email protected]> Co-authored-by: Ruodi Lu <[email protected]> [None][chore] Add failed cases into waives.txt (NVIDIA#9588) Signed-off-by: xinhe-nv <[email protected]> [https://nvbugs/5702793][fix] Fix uncontiguous tensor view (NVIDIA#9576) Signed-off-by: shuyix <[email protected]> [None][infra] Waive failed cases for main branch (NVIDIA#9615) Signed-off-by: qqiao <[email protected]> [TRTLLM-9488][feat] use FlashInfer.sampling by default (NVIDIA#9545) Signed-off-by: ixlmar <[email protected]> [None][infra] Update allowlist 2025/12/01 (NVIDIA#9616) Signed-off-by: Yuanjing Xue <[email protected]> [None][infra] Remove an invalid test name in waives.txt (NVIDIA#9620) Signed-off-by: qqiao <[email protected]> Lock the gpu clocks in L0 perf tests (NVIDIA#9585) Signed-off-by: Eran Geva <[email protected]> [TRTLLM-9466][test] Evaluate helix parallelism with DSV3 Lite (NVIDIA#9597) Signed-off-by: Balaram Buddharaju <[email protected]> [None][fix] Extract GPU count from single-node stage names (NVIDIA#9599) Signed-off-by: Chang Liu (Enterprise Products) <[email protected]> [https://nvbugs/5667774][fix] Refine Piecewise Cuda Graph Condition for DP (NVIDIA#9393) Signed-off-by: Jin Li <[email protected]> [TRTLLM-9144][fix] enhance RPC robustness (NVIDIA#8711) Signed-off-by: Superjomn <[email protected]> Signed-off-by: Erin Ho <[email protected]> Signed-off-by: Yan Chunwei <[email protected]> Co-authored-by: Erin Ho <[email protected]> [https://nvbugs/5627710][fix] Fix synchronization bugs in KvCacheTransferManager that can cause corrupted blocks (NVIDIA#9056) Signed-off-by: thorjohnsen <[email protected]> Signed-off-by: Thor Johnsen <[email protected]> Co-authored-by: Iman Tabrizian <[email protected]> Co-authored-by: Robin Kobus <[email protected]> [TRTLLM-8980][test] Clean up spec dec tests in test_llm_api_pytorch (NVIDIA#8889) Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [NVIDIA#9150][feat] Add code for nano v3 to custom implementation in AD (NVIDIA#9465) * Why? We would like to show an alternative to monkey-patching in AutoDeploy. * What? This commit builds on the existing custom model implementation for NemotronH and adds the bits relevant for MoE layers. Part of NVIDIA#9150. Signed-off-by: William Zhang <[email protected]> [NVIDIA#9150][feat] AutoDeploy: reviewer comments for NVIDIA#9150 (NVIDIA#9527) Signed-off-by: Lucas Liebenwein <[email protected]> [https://nvbugs/5651854][fix] Fix dist-serving perf by clearing CPU affinity (NVIDIA#9549) Signed-off-by: Shixiaowei02 <[email protected]> [NVIDIA#9550][feat] AutoDeploy: Add NVFP4 Cutlass MoE kernels (NVIDIA#9551) Signed-off-by: Neta Zmora <[email protected]> [https://nvbugs/5688388][fix] fix: Reducing num request in disagg test to speed up (NVIDIA#9598) Signed-off-by: Patrice Castonguay <[email protected]> [TRTLLM-8946][feat] Improved heuristics to detect shardable regions (NVIDIA#9200) Signed-off-by: Lucas Liebenwein <[email protected]> Signed-off-by: greg-kwasniewski1 <[email protected]> Co-authored-by: Lucas Liebenwein <[email protected]> [NVIDIA#9632][feat] Support EXTRA_WHEEL_BUILD_ARGS during wheel build (NVIDIA#9633) Signed-off-by: Yu Chi Li <[email protected]> [None][chore] Waive test failing on pre-merge (NVIDIA#9638) Signed-off-by: Balaram Buddharaju <[email protected]> [None][chore] Remove traceback dump for multimodal input processor (NVIDIA#9634) Signed-off-by: Chang Liu (Enterprise Products) <[email protected]> [None][chore] Fix trtllm-eval and move GroupedGemmInputsHelper (NVIDIA#9612) Signed-off-by: Enwei Zhu <[email protected]> [https://nvbugs/5698434][fix] Use separate weight mapper for draft (NVIDIA#9607) Signed-off-by: Anurag Mukkara <[email protected]> [TRTLLM-7101][infra] Reuse passed tests (NVIDIA#6894) Signed-off-by: Yiqing Yan <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> [None][test] Remove duplicate test cases (NVIDIA#9623) Signed-off-by: yufeiwu <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [None][feat] Add RocketKV usage doc and e2e accuracy test on LongBenchV2 (NVIDIA#9572) Signed-off-by: yuhangh <[email protected]> [TRTLLM-9242][doc] Add examples showcasing openai compatible APIs (NVIDIA#9520) Signed-off-by: Junyi Xu <[email protected]> [None][chore] AutoDeploy update cuda stream manager for multi-device (NVIDIA#9575) Signed-off-by: Suyog Gupta <[email protected]> [TRTLLM-9391][chore] Automatically estimate required workspace. (NVIDIA#9535) Signed-off-by: Bo Li <[email protected]> [https://nvbugs/5708475][fix] Fix e2e eval accuracy for helix parallelism (NVIDIA#9647) Signed-off-by: Balaram Buddharaju <[email protected]> [https://nvbugs/5561153][test] Fix log error for perf test (NVIDIA#9622) Signed-off-by: FredricZ-2007 <[email protected]> [TRTLLM-8241][feat] Aliasing to comply to LlmArgs (NVIDIA#9586) Signed-off-by: Pengyun Lin <[email protected]> [None][chore] Add failed cases into waives.txt (NVIDIA#9593) Signed-off-by: Jie Li <[email protected]> Co-authored-by: Jie Li <[email protected]> [TRTLLM-6842][feat] Support Response API for general purpose (NVIDIA#9392) Signed-off-by: Junyi Xu <[email protected]> [None][test] Update Qwen3-next accuracy testing by setting the cuda … (NVIDIA#9613) Signed-off-by: nv-guomingz <[email protected]> [None][feat] update trtllm-gen nvfp4 kernels with better performance (NVIDIA#9510) Signed-off-by: Perkz Zheng <[email protected]> [None][doc] Replace the tensorrt icon with torch icon on overview.md (NVIDIA#9644) Signed-off-by: nv-guomingz <[email protected]> [https://nvbugs/5705197][chore] Unwaive timeout disagg tests (NVIDIA#9637) Signed-off-by: Patrice Castonguay <[email protected]> [https://nvbugs/5552132][fix] Enable LoRa for GPT OSS Torch (NVIDIA#8253) Signed-off-by: Michal Guzek <[email protected]> [None][fix] Fix wide ep MoE error (NVIDIA#9642) Signed-off-by: Iman Tabrizian <[email protected]> [https://nvbugs/5702795][fix] Remove the warning message for aten.log. (NVIDIA#9665) Signed-off-by: nv-guomingz <[email protected]> [https://nvbugs/5693853][fix] Fix error handling when querying machin… (NVIDIA#9483) Signed-off-by: Gal Hubara Agam <[email protected]> [OMNIML-2932] [feat] nvfp4 awq support (NVIDIA#8698) Signed-off-by: weimingc <[email protected]> [NVIDIA#9643][fix] AutoDeploy: fix nano sharding config (NVIDIA#9668) Signed-off-by: Lucas Liebenwein <[email protected]> [NVIDIA#9147][feat] AutoDeploy: Draft Target Speculative Decoding (NVIDIA#9275) Signed-off-by: Govind Ramnarayan <[email protected]> [None][feat] Update Qwen3CodeToolParser to align tool-calling parameters (NVIDIA#9540) Signed-off-by: Wanli Jiang <[email protected]> [TRTLLM-7181][infra] Generate test results when pytest timeout happens (NVIDIA#9396) Signed-off-by: Yiqing Yan <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [TRTLLM-9522][fix] restore `trtllm-serve mm_embedding_serve` (NVIDIA#9669) [TRTLLM-5093][infra] Write env variables to a file in the interactive debug session (NVIDIA#6792) Signed-off-by: Yiqing Yan <[email protected]> [None][fix] fix error when processing batches containing both text and mm data (NVIDIA#8381) Signed-off-by: Nekofish-L <[email protected]> [TRTLLM-7073][feat] Support torch compile for PP for Llama and DeepSeekV3 (NVIDIA#7838) Signed-off-by: Jin Li <[email protected]> [None][feat] Add weights initialization and context phase parser to layer-wise benchmarks (NVIDIA#9667) Signed-off-by: Tailing Yuan <[email protected]> [TRTLLM-8274][feat] Check if executor is shutdown in /health entrypoint (NVIDIA#9057) Signed-off-by: Junyi Xu <[email protected]> [NVIDIA#8733][feat] Add Llama4 MoE handling to AutoDeploy (NVIDIA#9556) Signed-off-by: Tal Cherckez <[email protected]> Signed-off-by: tcherckez-nvidia <[email protected]> Co-authored-by: Neta Zmora <[email protected]> [None][ci] unwaive tests (NVIDIA#9651) Signed-off-by: Yan Chunwei <[email protected]> [None][feat] Add NIXL-LIBFABRIC support (NVIDIA#9225) Signed-off-by: Yoray Zack <[email protected]> Signed-off-by: zackyoray <[email protected]> [None][test] rename wide ep and disagg metric name in perf test (NVIDIA#9704) Signed-off-by: Ruodi Lu <[email protected]> Co-authored-by: Ruodi Lu <[email protected]> [https://nvbugs/5467531][fix] Unwaive fused_moe all to all test with … (NVIDIA#9617) Signed-off-by: Jin Li <[email protected]> [None][fix] Recover TRTLLM MoE Perf for DEP (NVIDIA#9562) Signed-off-by: Anthony Chang <[email protected]> [None][chore] Add failed cases into waives.txt (NVIDIA#9662) Signed-off-by: Xin He (SW-GPU) <[email protected]> Signed-off-by: xinhe-nv <[email protected]> Signed-off-by: Yanchao Lu <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> [None][fix] Fix TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS for MTP/EAGLE (NVIDIA#9608) Signed-off-by: Aurelien Chartier <[email protected]> [None][infra] Add container notices and documentation (NVIDIA#9185) Signed-off-by: Parker Drake <[email protected]> [TRTLLM-5312][infra] Add triton trigger rules (NVIDIA#6440) Signed-off-by: Yiqing Yan <[email protected]> [None][doc] Add feature docs for helix parallelism (NVIDIA#9684) Signed-off-by: Balaram Buddharaju <[email protected]> [TRTLLM-9579][infra] Set mergeWaiveList stage UNSTABLE when there is any issue (NVIDIA#9692) Signed-off-by: Yiqing Yan <[email protected]> [None][doc] Added line about partial reuse (NVIDIA#7846) Signed-off-by: thorjohnsen <[email protected]> [TRTLLM-8920][feat] decouple disagg service from fastapi (NVIDIA#8714) Signed-off-by: Lizhi Zhou <[email protected]> [https://nvbugs/5633340][fix] start disagg workers and servers on free ports (NVIDIA#9694) Signed-off-by: Lizhi Zhou <[email protected]> [TRTLLM-9562] [doc] Add Deployment Guide for Kimi K2 Thinking on TensorRT LLM - Blackwell (NVIDIA#9711) Signed-off-by: Kaiyu Xie <[email protected]> [NVIDIA#9602][feat] AutoDeploy: Support TRTLLM Sampler (NVIDIA#9641) Signed-off-by: Govind Ramnarayan <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [None] [tests] Unwaive EPLB tests (NVIDIA#9625) Signed-off-by: Kaiyu Xie <[email protected]> [https://nvbugs/5518713][test] Refactor core test lists by merging with llm_perf_cluster.yml (NVIDIA#9714) Signed-off-by: yufeiwu <[email protected]> [TRTLLM-7136][feat] Update load_weights method to include mapping parameter in checkpoint loaders (NVIDIA#9583) Signed-off-by: Robin Kobus <[email protected]> [None][refactor] Improve request processing function in sampler (NVIDIA#9671) Signed-off-by: Robin Kobus <[email protected]> [https://nvbugs/5670672][fix] Fix flaky KV connector tests (NVIDIA#9676) Signed-off-by: jthomson04 <[email protected]> [None][infra] Update allowed list 20251204 (NVIDIA#9718) Signed-off-by: Yuanjing Xue <[email protected]> [None][feat] AutoDeploy: Perf optimization for Attention and rmsnorm (NVIDIA#9719) Signed-off-by: Chenghao Zhang <[email protected]> [None][chore] Waive flakey disagg tests (NVIDIA#9749) Signed-off-by: Mike Iovine <[email protected]> [https://nvbugs/5601682][fix] Fix cacheTransceiver hang (NVIDIA#9311) Signed-off-by: Iman Tabrizian <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9199][docs] KV Connector Docs (NVIDIA#9325) Signed-off-by: jthomson04 <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9160][doc] add doc to llm_runtime.py (NVIDIA#9482) Signed-off-by: Yan Chunwei <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [None][doc] VDR 1.0 trtllm-serve doc enhancement (NVIDIA#9443) Signed-off-by: Pengyun Lin <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9086][doc] Clean up TODOs in documentation (NVIDIA#9292) Signed-off-by: junq <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9157][doc] Guided decoding doc improvement (NVIDIA#9359) Signed-off-by: Enwei Zhu <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [None][infra] Updated Linux installation guide (NVIDIA#9485) Signed-off-by: Yiqing Yan <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9075][doc] refine the slurm examples (NVIDIA#9548) Signed-off-by: Yan Chunwei <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9093][doc] update hyper links in overview (NVIDIA#9568) Signed-off-by: junq <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [TRTLLM-9092][doc] link to modelopt checkpoints in quick start guide (NVIDIA#9571) Signed-off-by: junq <[email protected]> Signed-off-by: Mike Iovine <[email protected]> Signed-off-by: Mike Iovine <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [None][fix] Fix triton moe load_weight (NVIDIA#9649) Signed-off-by: shuyix <[email protected]> [None][fix] fix a bug: deepseek_fp8_block_scales in TRTLLMGEN-MoE use 2D x_sf instead of 1D (NVIDIA#9658) Signed-off-by: xxi <[email protected]> [TRTLLM-9372][feat] Enable CuteDSL MoE with Large EP (NVIDIA#9592) Signed-off-by: Enwei Zhu <[email protected]> [TRTLLM-9522][chore] implement default `attach_multimodal_embeddings` (NVIDIA#9664) Signed-off-by: ixlmar <[email protected]> [TRTLLM-9660][feat] Convert cuteDSL GEMM to opt-in feature (NVIDIA#9682) Signed-off-by: Jonas Li <[email protected]> Co-authored-by: Kaiyu Xie <[email protected]> [None][fix] enable hmac in RPC (NVIDIA#9745) Signed-off-by: Superjomn <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [https://nvbugs/5703953][fix] Preserving ip:port for trtllm-serve before initializing llm (NVIDIA#9646) Signed-off-by: Junyi Xu <[email protected]> [None][infra] Waive failed cases for main branch on 12/07 (NVIDIA#9769) Signed-off-by: qqiao <[email protected]> [None][fix] Several minor fixes to CI setting (NVIDIA#9765) Signed-off-by: Yanchao Lu <[email protected]> [OMNIML-3036][doc] Re-branding TensorRT-Model-Optimizer as Nvidia Model-Optimizer (NVIDIA#9679) Signed-off-by: Chenjie Luo <[email protected]> [None][feat] Enable NCCL_SYMMETRIC as default fallback for AllReduce (NVIDIA#9314) Signed-off-by: Ludwig Schneider <[email protected]> [TRTLLM-9000][feat] Add multi-node Perf Tests into CI (NVIDIA#8800) Signed-off-by: Chenfei Zhang <[email protected]> [None][test] add ntp tolerance in time metrics verification (NVIDIA#9741) Signed-off-by: zhengd-nv <[email protected]> [TRTLLM-9603][feat] Enable ConfigurableMoE test in the CI (NVIDIA#9645) [https://nvbugs/5422621][test] Add GB 200 WIDEEP test case for RCCA 5422621 (NVIDIA#9506) Signed-off-by: FredricZ-2007 <[email protected]> [None][fix] Fix two tuning cache miss issues. (NVIDIA#9743) Signed-off-by: Yukun He <[email protected]> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <[email protected]> [TRTLLM-9706] [doc] Update wide EP documents (NVIDIA#9724) Signed-off-by: Kaiyu Xie <[email protected]> [https://nvbugs/5666804][test] only adding sampler config for limited models (NVIDIA#9512) Signed-off-by: Ruodi Lu <[email protected]> Co-authored-by: Ruodi Lu <[email protected]> Co-authored-by: yufeiwu-nv <[email protected]> Co-authored-by: Larry Xu <[email protected]> [None][infra] Waive failed cases for main on 12/08 (NVIDIA#9773) Signed-off-by: qqiao <[email protected]> [None][chore] Move the rocketkv e2e test to post-merge (NVIDIA#9768) Signed-off-by: Fanrong Li <[email protected]> [None][chore] Enable tvm_ffi for cute dsl nvfp4_gemm to reduce host overhead. (NVIDIA#9690) Signed-off-by: Mindy Li <[email protected]> [TRTLLM-9431][perf] Enable multistream for Linear Attention in Qwen3-… (NVIDIA#9696) Signed-off-by: nv-guomingz <[email protected]> [None][chore] Remove closed bugs (NVIDIA#9770) Signed-off-by: xinhe-nv <[email protected]> [None][infra] update mooncake in docker images (NVIDIA#9584) Signed-off-by: zhengd-nv <[email protected]> Signed-off-by: Zheng Duan <[email protected]> [None][test] Add Kimi k2 WIDEEP perf and accuracy cases (NVIDIA#9686) Signed-off-by: FredricZ-2007 <[email protected]> Signed-off-by: Kaiyu Xie <[email protected]> Co-authored-by: Kaiyu Xie <[email protected]> [https://nvbugs/5527655][test] Add test case for RCCA 5527655 (NVIDIA#9511) Signed-off-by: FredricZ-2007 <[email protected]> [http://nvbugs/5649010][fix] fix test_auto_scaling.py::test_worker_restart timeout (NVIDIA#9775) Signed-off-by: Lizhi Zhou <[email protected]> [None][fix] Switch AutoDeploy's default allreduce strategy to NCCL (NVIDIA#9666) Signed-off-by: Eran Geva <[email protected]> [TRTLLM-9506][fix] Fix AR for DeepSeek-R1 2 model path (NVIDIA#9661) Signed-off-by: qgai <[email protected]> ray + updatew works trtllm works in async env trtllm works in sync and async env ray + updatew works rebase to the updated verl server mode still cherry pick still cherry pick still cherry pick integrated http interface hang at RyExecutor create workers ray.remote clean code use tensorrt_llm.rlhf_utils Signed-off-by: Liwei Ma <[email protected]> placement, asyncllm, and basic tests Signed-off-by: Erin Ho <[email protected]> connect sleep and wakeup; Add support to pass None to update_weights Signed-off-by: Erin Ho <[email protected]> Batching ctx for IFB scheduler Signed-off-by: Yuan Tong <[email protected]> accuracy WAR for TP>1: always use AllReduceStrategy.NCCL, refactored Signed-off-by: Erin Ho <[email protected]> fix e2e integration Signed-off-by: Superjomn <[email protected]> update asyncllm, other nits Signed-off-by: Erin Ho <[email protected]> fix init setup Signed-off-by: Erin Ho <[email protected]> Fix TRTLLMSampler logprobs perf Signed-off-by: Yuan Tong <[email protected]> fix and cleanup Signed-off-by: Erin Ho <[email protected]> fix server Signed-off-by: Erin Ho <[email protected]> Revert "Batching ctx for IFB scheduler" This reverts commit b51aac0 Signed-off-by: Yuan Tong <[email protected]> update & address comments Signed-off-by: Erin Ho <[email protected]>
codego7250
pushed a commit
to codego7250/TensorRT-LLM
that referenced
this pull request
Dec 11, 2025
…port TRTLLMGenFusedMoE as backend (NVIDIA#9486)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR is part of the MoE refactoring task.
The main aspects of this PR are as follows:
I have conducted the functional test locally. However, before switching to ConfigurableMoE, we need to perform more performance tests to check for any performance regression.
This PR introduces an environment variable ENABLE_CONFIGURABLE_MOE. A value of 0 indicates that create_moe will use the original XXFusedMoE, while a value of 1 indicates that create_moe will use ConfigurableMoE. Currently, we only support "TRTLLM" to utilize ConfigurableMoE.
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
Details
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-listparameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.mdand the
scripts/test_to_stage_mapping.pyhelper.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.
Summary by CodeRabbit
Release Notes
New Features
Improvements
✏️ Tip: You can customize this high-level summary in your review settings.