[NVIDIA] Add MTP Benchmark Support for DeepSeek-R1 TRT - #392
Conversation
- Add dsr1_fp4_b200_trt_mtp_slurm.sh with MTP support - Add dsr1_fp8_b200_trt_mtp_slurm.sh with MTP support - Add dsr1_fp8_h200_trt_mtp_slurm.sh with MTP support - Refactored to use benchmark_lib.sh utilities - Use wait_for_server_ready and run_benchmark_serving functions
- Extended benchmark_lib.sh run_benchmark_serving() to support optional --use-chat-template flag - Added --use-chat-template to all three MTP benchmark scripts - This is required for MTP mode to work correctly
- Add dsr1-fp4-b200-trt-mtp configuration with EP/DP_ATTN/MTP logic - Add dsr1-fp8-b200-trt-mtp configuration with EP/DP_ATTN/MTP logic - Add dsr1-fp8-h200-trt-mtp configuration with EP/DP_ATTN/MTP logic - Configurations align with benchmark script logic for dynamic EP_SIZE, MOE_BACKEND, and MTP values
…vars - Remove duplicate EP_SIZE/DP_ATTENTION calculation logic from MTP scripts - MTP scripts now receive EP_SIZE and DP_ATTENTION as env vars from YAML config (like non-MTP scripts) - Only calculate MOE_BACKEND and MTP values based on DP_ATTENTION flag - Simplifies scripts from 156/117/112 lines to 104 lines each - Eliminates redundant logic between YAML configs and bash scripts
- Fix dsr1-fp4-b200-trt-mtp conc ranges to match EP_SIZE conditions - Fix dsr1-fp8-b200-trt-mtp conc ranges to match DP_ATTENTION conditions - Fix dsr1-fp8-h200-trt-mtp conc ranges to match DP_ATTENTION conditions - All configurations now accurately reflect the original bash script conditional logic
- Change conc-end from 64 to 32 for dsr1-fp8-b200-trt-mtp ISL=1024/OSL=1024 - Change conc-end from 128 to 64 for dsr1-fp8-b200-trt-mtp ISL=1024/OSL=8192 - Change conc-end from 64 to 32 for dsr1-fp8-b200-trt-mtp ISL=8192/OSL=1024 - Change conc-end from 256 to 128 for dsr1-fp8-h200-trt-mtp ISL=1024/OSL=8192 - Change conc-end from 64 to 32 for dsr1-fp8-h200-trt-mtp ISL=8192/OSL=1024 - All concurrency ranges now align to powers of 2: 4, 8, 16, 32, 64, 128, 256
- Remove overlapping boundaries between conc ranges - Change ranges to avoid overlap: 4-8, 16-64, 128-256 (with gaps at 9-15, 65-127) - All ranges now use powers of 2 boundaries without overlap - Applies to all TP/ISL/OSL combinations in dsr1-fp4-b200-trt-mtp
Summary of ChangesHello @lishicheng1996-nv, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly expands the benchmarking capabilities for DeepSeek-R1 models on TensorRT-LLM by integrating Multi-Token Prediction (MTP) support. It introduces new configurations and dedicated scripts to thoroughly evaluate MTP performance across different precisions and hardware, while also adding a Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds Multi-Token Prediction (MTP) benchmark support for DeepSeek-R1 on TensorRT-LLM. This includes new benchmark configurations, three new SLURM benchmark scripts, and an update to the benchmark library to support chat templates. The changes are generally well-structured. My review focuses on improving maintainability by reducing code duplication in the new benchmark scripts, ensuring consistency across them, and pointing out a potential configuration error that should be verified.
| - { tp: 8, ep: 8, conc-start: 4, conc-end: 32 } | ||
| - { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 256 } |
There was a problem hiding this comment.
The search-space defined here for isl: 8192, osl: 1024 is identical to the one for isl: 1024, osl: 1024 (lines 191-192). Given the significantly larger input sequence length, the supported concurrency range would likely be smaller due to higher memory usage. This might be a copy-paste error. Please verify if these values are correct. For reference, the dsr1-fp4-b200-trt-mtp configuration in this same file uses different concurrency ranges for different input sequence lengths.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
@cquil11 @functionstackx any idea why runs are skipped? |
|
@ankursingh-nv I was allowing the merge runs to finish |
|
You are talking about Run Sweep - [AMD] feat: Add mi355x distributed inference test CI workflow, right? Should I try again once it's completed? Ideally, these jobs should be queued. Why is that not the case? |
|
@cquil11 hey, can you please review and merge this PR? |
fix perf-changelog file
|
@functionstackx and @cquil11 where are we on this one? Any chance we can get it merged before EOD today? |
|
@cquil11 ^ |
|
@cquil11 Hey, any updates? It's almost a week now. Is there anything that we can do to expedite it? |
cquil11
left a comment
cquil11
left a comment
There was a problem hiding this comment.
lgtm
one comment but we can address that ad hoc
| PORT=$(( 8888 + $PORT_OFFSET )) | ||
| EXTRA_CONFIG_FILE="dsr1-fp8-mtp.yml" | ||
|
|
||
| # If ISL=8192 and DP_ATTENTION=true, export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:8192 |
There was a problem hiding this comment.
cquil11
changed the title
…) recipes (#1739) * minimaxm3 H200+H100 MTP: day-zero MiniMax-M3 EAGLE3 recipes Adds the spec-decoding=mtp siblings of the day-zero H200/H100 recipes (PR #1731): same MXFP8 target and serve shape, plus the Inferact/MiniMax-M3-EAGLE3 draft head via --speculative-config (method eagle3, 3 speculative tokens). The drafter is pinned to FLASH_ATTN — the EAGLE3 head is MHA and FlashInfer only supports the mandatory page size 128 through its GQA-only trtllm-gen kernel (the failure hit on the B300 MTP canary). Cudagraph capture scales to CONC * (1 + spec tokens); benchmark prompts run through the chat template so acceptance reflects real text. Search spaces mirror the non-MTP entries trimmed at the extreme-concurrency end (dsv4 / minimaxm3 b300-mtp precedent); H100 stays TP8-only with DEP omitted. Also adds SPEC_SUFFIX to the three H100 launchers (cw, cr, dgxc-slurm), which hardcoded _h100.sh and never gained the _mtp routing the H200 launchers have carried since #392 — without this, an mtp config on H100 silently runs the non-MTP script. This also fixes the latent same-bug for the existing qwen3.5-fp8-h100-sglang-mtp config. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf-changelog: fill in PR link for minimaxm3 H200/H100 MTP Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
5 participants
Adds Multi-Token Prediction (MTP) benchmarks for DeepSeek-R1 on TensorRT-LLM.
New Files (3):
Modified:
Note
Introduces MTP-based TRT experiments for DeepSeek-R1 and wires runners/benchmarks to use them.
dsr1-fp4-b200-trt-mtp,dsr1-fp8-b200-trt-mtp,dsr1-fp8-h200-trt-mtpinnvidia-master.yamlwithspec-decoding: mtpand updated TP/EP/DP-ATTN search spacesbenchmarks/*_trt_mtp_slurm.shscripts set MOE backend per platform, chooseMTP=1whendp-attnandMTP=3otherwise, computemax_num_tokens, and pass--use-chat-templaterun_benchmark_servingadds optional--use-chat-templateand builds command dynamicallySPEC_SUFFIXto route to*_mtp_slurm.shwhenSPEC_DECODING=mtpperf-changelog.yamldocumenting new MTP TRT supportWritten by Cursor Bugbot for commit c4cbd04. This will update automatically on new commits. Configure here.