Skip to content

Add GLM-5 NVFP4 GB200 disaggregated Dynamo TRT-LLM STP benchmarks / 新增 GLM-5 NVFP4 GB200 分离式 Dynamo TRT-LLM STP 基准测试 #1803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cquil11 merged 16 commits into from
Jul 15, 2026
Merged

Add GLM-5 NVFP4 GB200 disaggregated Dynamo TRT-LLM STP benchmarks / 新增 GLM-5 NVFP4 GB200 分离式 Dynamo TRT-LLM STP 基准测试 #1803

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
04e5a87
Add GLM-5.1 NVFP4 GB200 disaggregated TRT-LLM benchmarks via Dynamo (…
hshrivastava-droid Jul 8, 2026
d5d2b0d
Check in GLM5 trtllm_dynamo GB200 NVFP4 recipes locally + overlay at …
hshrivastava-droid Jul 8, 2026
911c9e3
Merge branch 'main' into rihuo/glm5-gb200-dynamo-trt
hshrivastava-droid Jul 8, 2026
cc6cba4
Switch glm5-fp4-gb200-dynamo-trt from GLM-5.1 to GLM-5
hshrivastava-droid Jul 9, 2026
041f77e
Merge branch 'main' into rihuo/glm5-gb200-dynamo-trt
hshrivastava-droid Jul 9, 2026
3239cdd
Point glm5-fp4-gb200-dynamo-trt MODEL_PATH at Lustre
hshrivastava-droid Jul 9, 2026
f998a3f
Merge branch 'main' into rihuo/glm5-gb200-dynamo-trt
hshrivastava-droid Jul 9, 2026
1d3da1f
Pin srt-slurm to v1.0.17 + adopt recipes/trtllm/glm5 overlay layout
hshrivastava-droid Jul 10, 2026
302e947
Merge branch 'main' into rihuo/glm5-gb200-dynamo-trt
hshrivastava-droid Jul 10, 2026
4b3c61c
fix: bump srt-slurm pin from v1.0.17 to v1.0.26
hshrivastava-droid Jul 13, 2026
ccdc2d7
Merge branch 'main' into rihuo/glm5-gb200-dynamo-trt
hshrivastava-droid Jul 13, 2026
af38b02
Merge branch 'main' into rihuo/glm5-gb200-dynamo-trt
cquil11 Jul 15, 2026
81b3fe3
Update perf-changelog.yaml
cquil11 Jul 15, 2026
117bf49
fix(config): declare NIXL for GLM-5 disaggregation
cquil11 Jul 15, 2026
03b11e3
Merge branch 'main' into rihuo/glm5-gb200-dynamo-trt
cquil11 Jul 15, 2026
1b1e56e
Update perf-changelog.yaml
cquil11 Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions ...sagg/trtllm_dynamo/gb200_nvfp4/ISL1K_OSL1K/STP/ctx1dep4_gen1dep32_batch16_eplb0_mtp0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
name: glm5_nvfp4_ISL1K_OSL1K_ctx1dep4_gen1dep32_batch16_eplb0_mtp0
model:
path: nvidia/GLM-5-NVFP4
container: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-dev.1-cuda13
precision: fp4
resources:
gpu_type: gb200
prefill_nodes: 1
prefill_workers: 1
gpus_per_prefill: 4
decode_workers: 1
decode_nodes: 8
gpus_per_decode: 32
gpus_per_node: 4
backend:
type: trtllm
prefill_environment:
ENROOT_ALLOW_DEV: 'yes'
MIMALLOC_PURGE_DELAY: '0'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_LOG_LEVEL: INFO
TRTLLM_ENABLE_PDL: '1'
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
decode_environment:
ENROOT_ALLOW_DEV: 'yes'
MIMALLOC_PURGE_DELAY: '0'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_LOG_LEVEL: INFO
TRTLLM_ENABLE_PDL: '1'
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
trtllm_config:
prefill:
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
enable_attention_dp: true
disable_overlap_scheduler: true
trust_remote_code: true
max_batch_size: 16
max_num_tokens: 16384
max_seq_len: 1064
print_iter_log: true
cuda_graph_config: null
moe_config:
backend: CUTEDSL
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
cache_transceiver_config:
backend: UCX
max_tokens_in_buffer: 16384
decode:
tensor_parallel_size: 32
moe_expert_parallel_size: 32
pipeline_parallel_size: 1
enable_attention_dp: true
enable_lm_head_tp_in_adp: false
trust_remote_code: true
max_batch_size: 16
max_num_tokens: 16
max_seq_len: 2088
print_iter_log: true
stream_interval: 100
num_postprocess_workers: 4
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
- 16
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.7
cache_transceiver_config:
backend: UCX
max_tokens_in_buffer: 16384
nvfp4_gemm_config:
allowed_backends:
- cutlass
- cublaslt
- cutedsl
- cuda_core
benchmark:
type: sa-bench
isl: 1024
osl: 1024
concurrencies: '666'
req_rate: 'inf'
num_prompts_mult: 16
use_chat_template: true
frontend:
type: dynamo
enable_multiple_frontends: true
health_check:
max_attempts: 360
interval_seconds: 10
dynamo:
install: false
111 changes: 111 additions & 0 deletions ...sagg/trtllm_dynamo/gb200_nvfp4/ISL1K_OSL1K/STP/ctx1dep4_gen1dep32_batch32_eplb0_mtp0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
name: glm5_nvfp4_ISL1K_OSL1K_ctx1dep4_gen1dep32_batch32_eplb0_mtp0
model:
path: nvidia/GLM-5-NVFP4
container: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-dev.1-cuda13
precision: fp4
resources:
gpu_type: gb200
prefill_nodes: 1
prefill_workers: 1
gpus_per_prefill: 4
decode_workers: 1
decode_nodes: 8
gpus_per_decode: 32
gpus_per_node: 4
backend:
type: trtllm
prefill_environment:
ENROOT_ALLOW_DEV: 'yes'
MIMALLOC_PURGE_DELAY: '0'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_LOG_LEVEL: INFO
TRTLLM_ENABLE_PDL: '1'
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
decode_environment:
ENROOT_ALLOW_DEV: 'yes'
MIMALLOC_PURGE_DELAY: '0'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_LOG_LEVEL: INFO
TRTLLM_ENABLE_PDL: '1'
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
trtllm_config:
prefill:
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
enable_attention_dp: true
disable_overlap_scheduler: true
trust_remote_code: true
max_batch_size: 16
max_num_tokens: 16384
max_seq_len: 1064
print_iter_log: true
cuda_graph_config: null
moe_config:
backend: CUTEDSL
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
cache_transceiver_config:
backend: UCX
max_tokens_in_buffer: 16384
decode:
tensor_parallel_size: 32
moe_expert_parallel_size: 32
pipeline_parallel_size: 1
enable_attention_dp: true
enable_lm_head_tp_in_adp: false
trust_remote_code: true
max_batch_size: 32
max_num_tokens: 32
max_seq_len: 2088
print_iter_log: true
stream_interval: 100
num_postprocess_workers: 4
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
- 16
- 24
- 32
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.7
cache_transceiver_config:
backend: UCX
max_tokens_in_buffer: 16384
nvfp4_gemm_config:
allowed_backends:
- cutlass
- cublaslt
- cutedsl
- cuda_core
benchmark:
type: sa-bench
isl: 1024
osl: 1024
concurrencies: '1229'
req_rate: 'inf'
num_prompts_mult: 16
use_chat_template: true
frontend:
type: dynamo
enable_multiple_frontends: true
health_check:
max_attempts: 360
interval_seconds: 10
dynamo:
install: false
171 changes: 171 additions & 0 deletions ...sagg/trtllm_dynamo/gb200_nvfp4/ISL1K_OSL1K/STP/ctx1dep4_gen1dep8_batch512_eplb0_mtp0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
name: glm5_nvfp4_ISL1K_OSL1K_ctx1dep4_gen1dep8_batch512_eplb0_mtp0
model:
path: nvidia/GLM-5-NVFP4
container: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-dev.1-cuda13
precision: fp4
resources:
gpu_type: gb200
prefill_nodes: 1
prefill_workers: 1
gpus_per_prefill: 4
decode_workers: 1
decode_nodes: 2
gpus_per_decode: 8
gpus_per_node: 4
backend:
type: trtllm
prefill_environment:
ENROOT_ALLOW_DEV: 'yes'
MIMALLOC_PURGE_DELAY: '0'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_LOG_LEVEL: INFO
TRTLLM_ENABLE_PDL: '1'
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
decode_environment:
ENROOT_ALLOW_DEV: 'yes'
MIMALLOC_PURGE_DELAY: '0'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_LOG_LEVEL: INFO
TRTLLM_ENABLE_PDL: '1'
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
trtllm_config:
prefill:
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
enable_attention_dp: true
disable_overlap_scheduler: true
trust_remote_code: true
max_batch_size: 16
max_num_tokens: 16384
max_seq_len: 1064
print_iter_log: true
cuda_graph_config: null
moe_config:
backend: CUTEDSL
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
cache_transceiver_config:
backend: UCX
max_tokens_in_buffer: 16384
decode:
tensor_parallel_size: 8
moe_expert_parallel_size: 8
pipeline_parallel_size: 1
enable_attention_dp: true
enable_lm_head_tp_in_adp: false
trust_remote_code: true
max_batch_size: 512
max_num_tokens: 512
max_seq_len: 2088
print_iter_log: true
stream_interval: 100
num_postprocess_workers: 4
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
- 16
- 24
- 32
- 40
- 48
- 56
- 64
- 72
- 80
- 88
- 96
- 104
- 112
- 120
- 128
- 136
- 144
- 152
- 160
- 168
- 176
- 184
- 192
- 200
- 208
- 216
- 224
- 232
- 240
- 248
- 256
- 264
- 272
- 280
- 288
- 296
- 304
- 312
- 320
- 328
- 336
- 344
- 352
- 360
- 368
- 376
- 384
- 392
- 400
- 408
- 416
- 424
- 432
- 440
- 448
- 456
- 464
- 472
- 480
- 488
- 496
- 504
- 512
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.8
cache_transceiver_config:
backend: UCX
max_tokens_in_buffer: 16384
nvfp4_gemm_config:
allowed_backends:
- cutlass
- cublaslt
- cutedsl
- cuda_core
benchmark:
type: sa-bench
isl: 1024
osl: 1024
concurrencies: '4301'
req_rate: 'inf'
num_prompts_mult: 16
use_chat_template: true
frontend:
type: dynamo
enable_multiple_frontends: true
health_check:
max_attempts: 360
interval_seconds: 10
dynamo:
install: false
Loading