Skip to content

[NV] Add GLM-5 NVFP4 GB200 disaggregated Dynamo TensorRT-LLM MTP benchmarks / [NV] 新增 GLM-5 NVFP4 GB200 分离式 Dynamo TensorRT-LLM MTP 基准测试 #1800

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adibarra merged 11 commits into from
Jul 16, 2026
Merged

[NV] Add GLM-5 NVFP4 GB200 disaggregated Dynamo TensorRT-LLM MTP benchmarks / [NV] 新增 GLM-5 NVFP4 GB200 分离式 Dynamo TensorRT-LLM MTP 基准测试 #1800

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f9f8f15
Add GLM-5 NVFP4 GB200 disaggregated TRT-LLM MTP benchmarks via Dynamo
hshrivastava-droid Jul 9, 2026
bfc8441
Update perf-changelog pr-link for #1800
hshrivastava-droid Jul 9, 2026
c9da830
Point glm5-fp4-gb200-dynamo-trt-mtp MODEL_PATH at Lustre
hshrivastava-droid Jul 9, 2026
5f0faec
Pin srt-slurm to v1.0.17 + adopt recipes/trtllm/glm5 overlay layout
hshrivastava-droid Jul 10, 2026
3d448df
Merge branch 'main' into rihuo/glm5-gb200-dynamo-trt-mtp
hshrivastava-droid Jul 10, 2026
8ebe723
update srt-slurm checkpoint
hshrivastava-droid Jul 14, 2026
cdaa3c9
Merge branch 'main' into rihuo/glm5-gb200-dynamo-trt-mtp
hshrivastava-droid Jul 14, 2026
5c491ab
Merge branch 'main' into rihuo/glm5-gb200-dynamo-trt-mtp
hshrivastava-droid Jul 15, 2026
851bd20
update kv p2p
hshrivastava-droid Jul 15, 2026
c92b4f4
Merge remote-tracking branch 'origin/main' into pr-1800
adibarra Jul 16, 2026
bf7bdd2
Merge remote-tracking branch 'origin/main' into pr-1800
adibarra Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 117 additions & 0 deletions ...sagg/trtllm_dynamo/gb200_nvfp4/ISL1K_OSL1K/MTP/ctx1dep4_gen1dep16_batch32_eplb0_mtp3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
name: glm5_nvfp4_ISL1K_OSL1K_ctx1dep4_gen1dep16_batch32_eplb0_mtp3
model:
path: nvidia/GLM-5-NVFP4
container: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-dev.1-cuda13
precision: fp4
resources:
gpu_type: gb200
prefill_nodes: 1
prefill_workers: 1
gpus_per_prefill: 4
decode_workers: 1
decode_nodes: 4
gpus_per_decode: 16
gpus_per_node: 4
backend:
type: trtllm
prefill_environment:
ENROOT_ALLOW_DEV: 'yes'
MIMALLOC_PURGE_DELAY: '0'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_LOG_LEVEL: INFO
TRTLLM_ENABLE_PDL: '1'
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
decode_environment:
ENROOT_ALLOW_DEV: 'yes'
MIMALLOC_PURGE_DELAY: '0'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_LOG_LEVEL: INFO
TRTLLM_ENABLE_PDL: '1'
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
trtllm_config:
prefill:
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
enable_attention_dp: true
disable_overlap_scheduler: true
trust_remote_code: true
max_batch_size: 16
max_num_tokens: 16384
max_seq_len: 1064
print_iter_log: true
cuda_graph_config: null
moe_config:
backend: CUTEDSL
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
cache_transceiver_config:
backend: UCX
max_tokens_in_buffer: 16384
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
decode:
tensor_parallel_size: 16
moe_expert_parallel_size: 16
pipeline_parallel_size: 1
enable_attention_dp: true
enable_lm_head_tp_in_adp: true
trust_remote_code: true
max_batch_size: 32
max_num_tokens: 128
max_seq_len: 2088
print_iter_log: true
stream_interval: 100
num_postprocess_workers: 4
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
- 16
- 24
- 32
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.7
cache_transceiver_config:
backend: UCX
max_tokens_in_buffer: 16384
nvfp4_gemm_config:
allowed_backends:
- cutlass
- cublaslt
- cutedsl
- cuda_core
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
benchmark:
type: sa-bench
isl: 1024
osl: 1024
concurrencies: '615'
req_rate: 'inf'
num_prompts_mult: 16
use_chat_template: true
frontend:
type: dynamo
enable_multiple_frontends: true
health_check:
max_attempts: 360
interval_seconds: 10
dynamo:
install: false
121 changes: 121 additions & 0 deletions ...sagg/trtllm_dynamo/gb200_nvfp4/ISL1K_OSL1K/MTP/ctx1dep4_gen1dep16_batch64_eplb0_mtp1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
name: glm5_nvfp4_ISL1K_OSL1K_ctx1dep4_gen1dep16_batch64_eplb0_mtp1
model:
path: nvidia/GLM-5-NVFP4
container: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-dev.1-cuda13
precision: fp4
resources:
gpu_type: gb200
prefill_nodes: 1
prefill_workers: 1
gpus_per_prefill: 4
decode_workers: 1
decode_nodes: 4
gpus_per_decode: 16
gpus_per_node: 4
backend:
type: trtllm
prefill_environment:
ENROOT_ALLOW_DEV: 'yes'
MIMALLOC_PURGE_DELAY: '0'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_LOG_LEVEL: INFO
TRTLLM_ENABLE_PDL: '1'
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
decode_environment:
ENROOT_ALLOW_DEV: 'yes'
MIMALLOC_PURGE_DELAY: '0'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_LOG_LEVEL: INFO
TRTLLM_ENABLE_PDL: '1'
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
trtllm_config:
prefill:
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
enable_attention_dp: true
disable_overlap_scheduler: true
trust_remote_code: true
max_batch_size: 16
max_num_tokens: 16384
max_seq_len: 1064
print_iter_log: true
cuda_graph_config: null
moe_config:
backend: CUTEDSL
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
cache_transceiver_config:
backend: UCX
max_tokens_in_buffer: 16384
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 1
decode:
tensor_parallel_size: 16
moe_expert_parallel_size: 16
pipeline_parallel_size: 1
enable_attention_dp: true
enable_lm_head_tp_in_adp: true
trust_remote_code: true
max_batch_size: 64
max_num_tokens: 128
max_seq_len: 2088
print_iter_log: true
stream_interval: 100
num_postprocess_workers: 4
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
- 16
- 24
- 32
- 40
- 48
- 56
- 64
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.7
cache_transceiver_config:
backend: UCX
max_tokens_in_buffer: 16384
nvfp4_gemm_config:
allowed_backends:
- cutlass
- cublaslt
- cutedsl
- cuda_core
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 1
benchmark:
type: sa-bench
isl: 1024
osl: 1024
concurrencies: '1229'
req_rate: 'inf'
num_prompts_mult: 16
use_chat_template: true
frontend:
type: dynamo
enable_multiple_frontends: true
health_check:
max_attempts: 360
interval_seconds: 10
dynamo:
install: false
115 changes: 115 additions & 0 deletions ...sagg/trtllm_dynamo/gb200_nvfp4/ISL1K_OSL1K/MTP/ctx1dep4_gen1dep32_batch16_eplb0_mtp3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
name: glm5_nvfp4_ISL1K_OSL1K_ctx1dep4_gen1dep32_batch16_eplb0_mtp3
model:
path: nvidia/GLM-5-NVFP4
container: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-dev.1-cuda13
precision: fp4
resources:
gpu_type: gb200
prefill_nodes: 1
prefill_workers: 1
gpus_per_prefill: 4
decode_workers: 1
decode_nodes: 8
gpus_per_decode: 32
gpus_per_node: 4
backend:
type: trtllm
prefill_environment:
ENROOT_ALLOW_DEV: 'yes'
MIMALLOC_PURGE_DELAY: '0'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_LOG_LEVEL: INFO
TRTLLM_ENABLE_PDL: '1'
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
decode_environment:
ENROOT_ALLOW_DEV: 'yes'
MIMALLOC_PURGE_DELAY: '0'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_LOG_LEVEL: INFO
TRTLLM_ENABLE_PDL: '1'
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
trtllm_config:
prefill:
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
enable_attention_dp: true
disable_overlap_scheduler: true
trust_remote_code: true
max_batch_size: 16
max_num_tokens: 16384
max_seq_len: 1064
print_iter_log: true
cuda_graph_config: null
moe_config:
backend: CUTEDSL
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
cache_transceiver_config:
backend: UCX
max_tokens_in_buffer: 16384
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
decode:
tensor_parallel_size: 32
moe_expert_parallel_size: 32
pipeline_parallel_size: 1
enable_attention_dp: true
enable_lm_head_tp_in_adp: true
trust_remote_code: true
max_batch_size: 16
max_num_tokens: 64
max_seq_len: 2088
print_iter_log: true
stream_interval: 100
num_postprocess_workers: 4
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
- 16
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
cache_transceiver_config:
backend: UCX
max_tokens_in_buffer: 16384
nvfp4_gemm_config:
allowed_backends:
- cutlass
- cublaslt
- cutedsl
- cuda_core
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
benchmark:
type: sa-bench
isl: 1024
osl: 1024
concurrencies: '666'
req_rate: 'inf'
num_prompts_mult: 16
use_chat_template: true
frontend:
type: dynamo
enable_multiple_frontends: true
health_check:
max_attempts: 360
interval_seconds: 10
dynamo:
install: false
Loading