[fix] Fix perf regression caused by MoE autotuner when using DeepEPLowLatency #6288
Conversation
jinyangyuan-nvidia
requested review from
hyukn,
nv-yilinf,
yilin-void and
yuantailing
|
/bot run |
📝 Walkthrough## Walkthrough
The changes update the MoE (Mixture of Experts) operator integration by removing the `enable_alltoall` parameter from the `MoERunner` constructor and hardcoding it to `False`. The `fused_moe` function now accepts optional tuning parameters to decouple tuning inputs from runtime inputs. The `forward_chunk` method passes these tuning parameters when all-to-all communication is used. Additionally, utility functions returning power-of-2 token buckets now return them in ascending order.
## Changes
| File(s) | Change Summary |
|-------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|
| tensorrt_llm/_torch/custom_ops/torch_custom_ops.py | Removed `enable_alltoall` from `MoERunner` constructor; added optional tuning parameters to `fused_moe`; adjusted input handling and assertions based on `enable_alltoall`. |
| tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py | In `forward_chunk`, defined and passed `tuner_num_tokens` and `tuner_top_k` to `fused_moe` when all-to-all is enabled. |
| tensorrt_llm/_torch/utils.py | Reversed the return order of `get_power_of_2_num_tokens_buckets` and `get_last_power_of_2_num_tokens_buckets` to ascending order. |
## Sequence Diagram(s)
```mermaid
sequenceDiagram
participant User
participant WideEPMoE
participant fused_moe
participant MoERunner
User->>WideEPMoE: forward_chunk(input, ...)
WideEPMoE->>WideEPMoE: Compute tuner_num_tokens, tuner_top_k if all_to_all
WideEPMoE->>fused_moe: Call with tuner_num_tokens, tuner_top_k
fused_moe->>MoERunner: Instantiate (enable_alltoall=False)
MoERunner-->>fused_moe: Return result
fused_moe-->>WideEPMoE: Return output
WideEPMoE-->>User: Return outputEstimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Suggested reviewers
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
tensorrt_llm/_torch/custom_ops/torch_custom_ops.py (1)
151-170: Verify the impact of zero-padding on tuning accuracyWhen
enable_alltoallis True and the input needs padding (lines 157-163), zeros are used to fill the gap. This could potentially affect the autotuner's tactic selection if the kernel behavior differs significantly between zero and non-zero inputs.Consider documenting why zero-padding is appropriate here, or if alternative padding strategies were considered.
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py (1)
440-451: Well-implemented parameter preparation for tuningThe logic correctly prepares
total_valid_tokensandoriginal_top_kbased on the alltoall configuration. The calculation handles both cases where rank token counts are provided or need to be inferred.Consider adding a brief comment explaining why these parameters are needed for tuning when alltoall is enabled:
# Prepare additional information for profiling in case padding is applied in all-to-all +# These parameters help the autotuner select appropriate tactics by providing +# consistent input dimensions regardless of actual runtime token distribution if use_all_to_all:
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
tensorrt_llm/_torch/custom_ops/torch_custom_ops.py(4 hunks)tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py(2 hunks)tensorrt_llm/_torch/utils.py(1 hunks)
🔇 Additional comments (3)
tensorrt_llm/_torch/custom_ops/torch_custom_ops.py (2)
57-58: Clarify the rationale for hardcodingenable_alltoallto FalseThe comment states "The best tactic is estimated as if alltoall is disabled", but it's not immediately clear why this assumption is made. This change appears to be central to fixing the performance regression. Could you provide more context on why disabling alltoall during tactic estimation resolves the issue?
189-210: Good separation of tuning and runtime parametersThe design effectively decouples the autotuner's tactic selection from runtime execution by using
tuner_inputandtuner_top_kfor profiling while preserving the original parameters for actual execution. This should help stabilize performance when using alltoall with varying input sizes.tensorrt_llm/_torch/utils.py (1)
235-242: Verify bucket ordering change in callers ofget_last_power_of_2_num_tokens_bucketsThe function now returns power-of-2 buckets in ascending order (small → large) instead of descending. This behavioral change may break any code that assumed the old ordering. We found usages in:
tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py(lines wherem_values = get_last_power_of_2_num_tokens_buckets(MAX_PROFILE_BUCKET))tensorrt_llm/_torch/custom_ops/torch_custom_ops.py(passed intoDynamicTensorSpec)Please confirm:
- The switch to ascending order is intentional and aligns with the MoE autotuner fix.
- All callers listed above have been audited and updated (e.g., iteration order, indexing, profiling logic) to handle ascending buckets correctly.
|
PR_Github #12687 [ run ] triggered by Bot |
|
PR_Github #12687 [ run ] completed with state |
jinyangyuan-nvidia
force-pushed
the
dev/fix_deepep_autotuner
branch
from
2e70ade to
ccb4a25
Compare
|
/bot run |
|
PR_Github #12763 [ run ] triggered by Bot |
|
PR_Github #12763 [ run ] completed with state |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py (1)
441-452: LGTM: Clear separation of tuning parameters from runtime parameters.The logic correctly prepares
tuner_num_tokensandtuner_top_kwhenuse_all_to_allis enabled, ensuring the tuner receives inputs equivalent to the non-alltoall case during warmup. The variable naming aligns with past feedback about clarifying the tuning-specific purpose.Note: The static analysis warning about line 443 being too long (147 > 120 characters) should be addressed for consistency with coding guidelines.
Apply this diff to fix the line length:
- # Therefore, to get the correct tactics during the actual inference, the inputs to the tuner should be the same as when not using alltoall. + # Therefore, to get the correct tactics during the actual inference, the inputs to the tuner + # should be the same as when not using alltoall.tensorrt_llm/_torch/custom_ops/torch_custom_ops.py (1)
151-161: LGTM: Robust parameter validation and input preparation.The logic correctly validates that tuning parameters are provided when
enable_alltoall=Trueand absent whenFalse. The input slicing fortuner_inputand derivation oftuner_top_kare appropriate for maintaining consistency between tuning and runtime phases.Note: The static analysis warning about line 152 being too long (143 > 120 characters) should be addressed.
Apply this diff to fix the line length:
- # Therefore, to get the correct tactics during the actual inference, the inputs to the tuner should be the same as when not using alltoall. + # Therefore, to get the correct tactics during the actual inference, the inputs to the tuner + # should be the same as when not using alltoall.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
tensorrt_llm/_torch/custom_ops/torch_custom_ops.py(4 hunks)tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py(2 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
**/*.py: Python code should conform to Python 3.8+.
Indent Python code with 4 spaces. Do not use tabs.
Always maintain the namespace when importing in Python, even if only one class or function from a module is used.
Python filenames should use snake_case (e.g., some_file.py).
Python classes should use PascalCase (e.g., class SomeClass).
Python functions and methods should use snake_case (e.g., def my_awesome_function():).
Python local variables should use snake_case. Prefix k for variable names that start with a number (e.g., k_99th_percentile).
Python global variables should use upper snake_case and prefix G (e.g., G_MY_GLOBAL).
Python constants should use upper snake_case (e.g., MY_CONSTANT).
Avoid shadowing variables declared in an outer scope in Python.
Initialize all externally visible members of a Python class in the constructor.
For interfaces that may be used outside a Python file, prefer docstrings over comments.
Comments in Python should be reserved for code within a function, or interfaces that are local to a file.
Use Google style docstrings for Python classes and functions, which can be parsed by Sphinx.
Attributes and variables in Python can be documented inline; attribute docstrings will be rendered under the docstring for the class.
Avoid using reflection in Python when functionality can be easily achieved without it.
When using try-except blocks in Python, limit the except to the smallest set of errors possible.
When using try-except blocks to handle multiple possible variable types in Python, keep the body of the try as small as possible, using the else block to implement the logic.
Files:
tensorrt_llm/_torch/custom_ops/torch_custom_ops.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
**/*.{cpp,h,hpp,cc,cxx,cu,py}
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
All TensorRT-LLM Open Source Software code should contain an NVIDIA copyright header that includes the current year. This includes .cpp, .h, .cu, .py, and any other source files which are compiled or interpreted.
Files:
tensorrt_llm/_torch/custom_ops/torch_custom_ops.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
🪛 Ruff (0.12.2)
tensorrt_llm/_torch/custom_ops/torch_custom_ops.py
152-152: Line too long (143 > 120)
(E501)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
443-443: Line too long (147 > 120)
(E501)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (5)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py (1)
685-686: LGTM: Consistent parameter passing to fused_moe.The
tuner_num_tokensandtuner_top_kparameters are correctly passed to thefused_moeoperator, maintaining consistency with the logic in the custom ops layer.tensorrt_llm/_torch/custom_ops/torch_custom_ops.py (4)
57-58: LGTM: Hardcoding enable_alltoall=False for consistent tuning.This change ensures that the MoE autotuner always profiles with
enable_alltoall=False, providing consistent tactics regardless of runtime communication patterns. The comment clearly explains the rationale.
144-145: LGTM: Well-designed optional parameters for tuning separation.The addition of
tuner_num_tokensandtuner_top_kas optional parameters provides a clean interface for separating tuning inputs from runtime inputs when all-to-all communication is enabled.
168-168: LGTM: Consistent use of tuning parameters.The MoERunner is correctly instantiated with
tuner_top_kinstead of the originaltoken_selected_experts.size(1), ensuring the tuner configuration matches the prepared tuning inputs.
186-187: LGTM: Proper separation of tuning and runtime inputs.The autotuner correctly uses
tuner_inputfor tactic selection while the actualrun_moecall uses the originalinputtensor andenable_alltoallflag. This separation addresses the performance regression by ensuring consistent tuning behavior.Also applies to: 197-198
|
/bot run |
|
PR_Github #13053 [ run ] triggered by Bot |
|
PR_Github #13053 [ run ] completed with state |
|
/bot run |
|
PR_Github #13063 [ run ] triggered by Bot |
|
PR_Github #13063 [ run ] completed with state |
|
/bot run |
|
PR_Github #13071 [ run ] triggered by Bot |
|
PR_Github #13071 [ run ] completed with state |
jinyangyuan-nvidia
force-pushed
the
dev/fix_deepep_autotuner
branch
from
5eb81a7 to
ede34be
Compare
|
/bot run |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py (1)
440-452: Good implementation of tuning parameter decoupling, but fix line length.The logic correctly decouples tuning inputs from runtime inputs when
use_all_to_allis enabled, ensuring consistent tactic selection between warmup and inference phases. The variable namingtuner_num_tokensappropriately reflects its purpose as discussed in past reviews.Apply this diff to fix the line length violation:
- # Prepare additional information for profiling in case padding is applied when using alltoall. + # Prepare additional information for profiling in case padding is applied + # when using alltoall.tensorrt_llm/_torch/custom_ops/torch_custom_ops.py (1)
151-162: Good parameter validation logic, but fix line length.The conditional logic correctly validates parameter usage and sets up appropriate tuning inputs. The assertions ensure proper API usage, and the input slicing for alltoall cases maintains profiling consistency.
Apply this diff to fix the line length violation:
- # Therefore, to get the correct tactics during the actual inference, the inputs to the tuner should be the same as when not using alltoall. + # Therefore, to get the correct tactics during the actual inference, the inputs + # to the tuner should be the same as when not using alltoall.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
tensorrt_llm/_torch/custom_ops/torch_custom_ops.py(4 hunks)tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py(2 hunks)tensorrt_llm/_torch/utils.py(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- tensorrt_llm/_torch/utils.py
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
**/*.py: Python code should conform to Python 3.8+.
Indent Python code with 4 spaces. Do not use tabs.
Always maintain the namespace when importing in Python, even if only one class or function from a module is used.
Python filenames should use snake_case (e.g., some_file.py).
Python classes should use PascalCase (e.g., class SomeClass).
Python functions and methods should use snake_case (e.g., def my_awesome_function():).
Python local variables should use snake_case. Prefix k for variable names that start with a number (e.g., k_99th_percentile).
Python global variables should use upper snake_case and prefix G (e.g., G_MY_GLOBAL).
Python constants should use upper snake_case (e.g., MY_CONSTANT).
Avoid shadowing variables declared in an outer scope in Python.
Initialize all externally visible members of a Python class in the constructor.
For interfaces that may be used outside a Python file, prefer docstrings over comments.
Comments in Python should be reserved for code within a function, or interfaces that are local to a file.
Use Google style docstrings for Python classes and functions, which can be parsed by Sphinx.
Attributes and variables in Python can be documented inline; attribute docstrings will be rendered under the docstring for the class.
Avoid using reflection in Python when functionality can be easily achieved without it.
When using try-except blocks in Python, limit the except to the smallest set of errors possible.
When using try-except blocks to handle multiple possible variable types in Python, keep the body of the try as small as possible, using the else block to implement the logic.
Files:
tensorrt_llm/_torch/custom_ops/torch_custom_ops.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
**/*.{cpp,h,hpp,cc,cxx,cu,py}
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
All TensorRT-LLM Open Source Software code should contain an NVIDIA copyright header that includes the current year. This includes .cpp, .h, .cu, .py, and any other source files which are compiled or interpreted.
Files:
tensorrt_llm/_torch/custom_ops/torch_custom_ops.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
🪛 Ruff (0.12.2)
tensorrt_llm/_torch/custom_ops/torch_custom_ops.py
152-152: Line too long (143 > 120)
(E501)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
443-443: Line too long (147 > 120)
(E501)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (4)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py (1)
685-686: LGTM! Correct usage of tuning parameters.The tuning parameters are properly passed to the
fused_moefunction, maintaining consistency with the conditional logic established earlier in the method.tensorrt_llm/_torch/custom_ops/torch_custom_ops.py (3)
57-58: LGTM! Correct hardcoding of enable_alltoall for consistent tuning.The hardcoding ensures that tactic selection during profiling always assumes alltoall is disabled, providing consistent performance optimization regardless of runtime alltoall usage. This aligns with the design intent discussed in past reviews.
144-145: LGTM! Appropriate parameter additions for tuning decoupling.The new optional parameters enable the separation of tuning inputs from runtime inputs when alltoall is used, with appropriate types and default values.
168-168: LGTM! Consistent usage of tuning parameters.The
tuner_top_kandtuner_inputvariables are used consistently in the MoERunner construction and tactic selection calls, maintaining the intended separation between tuning and runtime contexts.Also applies to: 186-188, 197-199
|
PR_Github #13097 [ run ] triggered by Bot |
|
PR_Github #13097 [ run ] completed with state |
…wLatency Signed-off-by: Jinyang Yuan <[email protected]>
Signed-off-by: Jinyang Yuan <[email protected]>
Signed-off-by: Jinyang Yuan <[email protected]>
Signed-off-by: Jinyang Yuan <[email protected]>
jinyangyuan-nvidia
force-pushed
the
dev/fix_deepep_autotuner
branch
from
ede34be to
d7f5461
Compare
|
/bot run |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py (1)
443-443: Fix line length violation.The comment exceeds the 120-character limit specified in the static analysis hints.
Apply this diff to fix the line length:
- # Therefore, to get the correct tactics during the actual inference, the inputs to the tuner should be the same as when not using alltoall. + # Therefore, to get the correct tactics during the actual inference, the inputs to the tuner + # should be the same as when not using alltoall.tensorrt_llm/_torch/custom_ops/torch_custom_ops.py (1)
151-161: Fix line length violation and approve the tuning input logic.The implementation correctly handles the separation of tuning and runtime inputs with proper parameter validation through assertions.
Apply this diff to fix the line length violation:
- # Therefore, to get the correct tactics during the actual inference, the inputs to the tuner should be the same as when not using alltoall. + # Therefore, to get the correct tactics during the actual inference, the inputs to the tuner + # should be the same as when not using alltoall.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
tensorrt_llm/_torch/custom_ops/torch_custom_ops.py(4 hunks)tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py(2 hunks)tensorrt_llm/_torch/utils.py(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- tensorrt_llm/_torch/utils.py
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
**/*.py: Python code should conform to Python 3.8+.
Indent Python code with 4 spaces. Do not use tabs.
Always maintain the namespace when importing in Python, even if only one class or function from a module is used.
Python filenames should use snake_case (e.g., some_file.py).
Python classes should use PascalCase (e.g., class SomeClass).
Python functions and methods should use snake_case (e.g., def my_awesome_function():).
Python local variables should use snake_case. Prefix k for variable names that start with a number (e.g., k_99th_percentile).
Python global variables should use upper snake_case and prefix G (e.g., G_MY_GLOBAL).
Python constants should use upper snake_case (e.g., MY_CONSTANT).
Avoid shadowing variables declared in an outer scope in Python.
Initialize all externally visible members of a Python class in the constructor.
For interfaces that may be used outside a Python file, prefer docstrings over comments.
Comments in Python should be reserved for code within a function, or interfaces that are local to a file.
Use Google style docstrings for Python classes and functions, which can be parsed by Sphinx.
Attributes and variables in Python can be documented inline; attribute docstrings will be rendered under the docstring for the class.
Avoid using reflection in Python when functionality can be easily achieved without it.
When using try-except blocks in Python, limit the except to the smallest set of errors possible.
When using try-except blocks to handle multiple possible variable types in Python, keep the body of the try as small as possible, using the else block to implement the logic.
Files:
tensorrt_llm/_torch/custom_ops/torch_custom_ops.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
**/*.{cpp,h,hpp,cc,cxx,cu,py}
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
All TensorRT-LLM Open Source Software code should contain an NVIDIA copyright header that includes the current year. This includes .cpp, .h, .cu, .py, and any other source files which are compiled or interpreted.
Files:
tensorrt_llm/_torch/custom_ops/torch_custom_ops.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
🪛 Ruff (0.12.2)
tensorrt_llm/_torch/custom_ops/torch_custom_ops.py
152-152: Line too long (143 > 120)
(E501)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
443-443: Line too long (147 > 120)
(E501)
🔇 Additional comments (6)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py (2)
440-452: Approve the tuning parameter preparation logic.The implementation correctly prepares tuning parameters to address the performance regression. When
use_all_to_allisTrue, it computestuner_num_tokensfrom either the sum ofall_rank_num_tokensor as a fallback usingx.shape[0] * self.mapping.tp_size, and setstuner_top_kfrom the token selection shape. Whenuse_all_to_allisFalse, both parameters are set toNone, ensuring the tuner uses the original logic.This approach ensures that the tuner receives consistent inputs regardless of the alltoall setting, which should resolve the performance regression mentioned in the PR objectives.
685-686: Approve the tuning parameter usage.The parameters are correctly passed to the
fused_moeoperator call, maintaining consistency with the preparation logic above and the expected interface changes in the custom ops module.tensorrt_llm/_torch/custom_ops/torch_custom_ops.py (4)
57-58: Approve the hardcoded enable_alltoall setting.Setting
self.enable_alltoall = Falsein theMoERunnerconstructor aligns with the PR's objective to ensure tuning always uses the non-alltoall path. This addresses the performance regression by ensuring consistent tuning behavior regardless of runtime alltoall usage.
144-145: Approve the new tuning parameters.The addition of
tuner_num_tokensandtuner_top_kas optional parameters provides the necessary interface for separating tuning inputs from runtime inputs, which is the core fix for the performance regression.
168-168: Approve the use of tuner_top_k for MoERunner creation.Using
tuner_top_kinstead of the original top_k parameter ensures the runner is configured correctly for tuning purposes, maintaining consistency with the tuning input preparation logic.
186-187: Approve the use of tuner_input for GEMM tactic selection.Using
tuner_inputfor both GEMM1 and GEMM2 tactic selection ensures that the autotuner receives consistent inputs regardless of the runtime alltoall setting, which should resolve the performance regression described in the PR objectives.Also applies to: 197-198
|
PR_Github #13127 [ run ] triggered by Bot |
|
PR_Github #13127 [ run ] completed with state |
…wLatency (NVIDIA#6288) Signed-off-by: Jinyang Yuan <[email protected]> Signed-off-by: Shreyas Misra <[email protected]>
…wLatency (NVIDIA#6288) Signed-off-by: Jinyang Yuan <[email protected]> Signed-off-by: Ransiki Zhang <[email protected]>
…wLatency (NVIDIA#6288) Signed-off-by: Jinyang Yuan <[email protected]> Signed-off-by: Lanyu Liao <[email protected]>
4 participants
Summary by CodeRabbit
New Features
Bug Fixes
Refactor
Description
Test Coverage
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-listparameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.mdand the
scripts/test_to_stage_mapping.pyhelper.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.