Skip to content

[https://nvbugs/5844149][fix] Fix issues with DSV3.2 perf tests #13054

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
brb-nv wants to merge 2 commits into from
Closed

[https://nvbugs/5844149][fix] Fix issues with DSV3.2 perf tests #13054

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b82c6c6
[https://nvbugs/5844149][fix] Fix OOM in ctx server and IMA in gen se…
brb-nv Apr 9, 2026
e0838e9
cleanup
brb-nv Apr 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions jenkins/scripts/perf/local/submit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,8 +286,6 @@ def generate_sbatch_params(args, hardware_config, work_dir):
f"#SBATCH --segment={total_nodes}",
f"#SBATCH --ntasks={total_gpus}",
f"#SBATCH --ntasks-per-node={gpus_per_node}",
f"#SBATCH --gpus-per-node={gpus_per_node}",
f"#SBATCH --gres=gpu:{gpus_per_node}",
f"#SBATCH --partition={args.partition}",
f"#SBATCH --time={args.time}",
f"#SBATCH --account={args.account}",
Expand Down Expand Up @@ -550,6 +548,13 @@ def main():
test_name=select_pattern,
)

if benchmark_mode in ("gen_only",):
config.setdefault("benchmark", {})["multi_round"] = 1
Comment thread
chenfeiz0326 marked this conversation as resolved.
modified_config_path = os.path.join(work_dir, os.path.basename(config_yaml))
with open(modified_config_path, "w") as f:
yaml.dump(config, f, default_flow_style=False, sort_keys=False)
config_yaml = modified_config_path

# Generate sbatch params
sbatch_lines = generate_sbatch_params(args, hardware_config, work_dir)

Expand Down
17 changes: 11 additions & 6 deletions ...saggregated/gb200_deepseek-v32-fp4_32k4k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ benchmark:
mode: e2e
use_nv_sa_benchmark: false
multi_round: 10
benchmark_ratio: 0.0
benchmark_ratio: 1.0
Comment thread
chenfeiz0326 marked this conversation as resolved.
streaming: true
concurrency_list: '1'
input_length: 32768
Expand Down Expand Up @@ -48,24 +48,27 @@ worker_config:
gen:
print_iter_log: true
max_batch_size: 1
max_num_tokens: 256
max_num_tokens: 8
max_input_len: 32784
max_seq_len: 40960
tensor_parallel_size: 8
moe_expert_parallel_size: 8
pipeline_parallel_size: 1
context_parallel_size: 1
enable_attention_dp: false
enable_lm_head_tp_in_adp: false
cuda_graph_config:
enable_padding: true
max_batch_size: 1
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.9
free_gpu_memory_fraction: 0.25
dtype: fp8
tokens_per_block: 64
moe_config:
backend: TRTLLM
cache_transceiver_config:
max_tokens_in_buffer: 32768
max_tokens_in_buffer: 32832
backend: UCX
disable_overlap_scheduler: true
speculative_config: &id001
Expand All @@ -84,6 +87,8 @@ worker_config:
print_iter_log: true
max_batch_size: 1
max_num_tokens: 32784
max_input_len: 32784
max_seq_len: 32832
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
Expand All @@ -93,14 +98,14 @@ worker_config:
cuda_graph_config: null
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
free_gpu_memory_fraction: 0.4
dtype: fp8
tokens_per_block: 64
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 32768
max_tokens_in_buffer: 32832
backend: UCX
disable_overlap_scheduler: true
speculative_config: *id001
12 changes: 8 additions & 4 deletions ...gated/gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-UCX.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ worker_config:
print_iter_log: true
max_batch_size: 64
max_num_tokens: 128
max_input_len: 32784
max_seq_len: 40960
tensor_parallel_size: 32
moe_expert_parallel_size: 32
pipeline_parallel_size: 1
Expand All @@ -60,7 +62,7 @@ worker_config:
max_batch_size: 64
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.85
free_gpu_memory_fraction: 0.5
dtype: fp8
tokens_per_block: 64
moe_config:
Expand All @@ -70,7 +72,7 @@ worker_config:
num_slots: 288
layer_updates_per_iter: 1
cache_transceiver_config:
max_tokens_in_buffer: 32768
max_tokens_in_buffer: 32832
backend: UCX
disable_overlap_scheduler: true
speculative_config: &id001
Expand All @@ -88,6 +90,8 @@ worker_config:
print_iter_log: true
max_batch_size: 1
max_num_tokens: 32784
max_input_len: 32784
max_seq_len: 32832
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
Expand All @@ -97,14 +101,14 @@ worker_config:
cuda_graph_config: null
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
free_gpu_memory_fraction: 0.4
dtype: fp8
tokens_per_block: 64
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 32768
max_tokens_in_buffer: 32832
backend: UCX
disable_overlap_scheduler: true
speculative_config: *id001
14 changes: 9 additions & 5 deletions ...gregated/gb200_deepseek-v32-fp4_32k4k_con256_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-UCX.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,9 @@ worker_config:
gen:
print_iter_log: true
max_batch_size: 8
max_num_tokens: 256
max_num_tokens: 64
max_input_len: 32784
max_seq_len: 40960
tensor_parallel_size: 32
moe_expert_parallel_size: 32
pipeline_parallel_size: 1
Expand All @@ -60,14 +62,14 @@ worker_config:
max_batch_size: 8
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.85
free_gpu_memory_fraction: 0.5
dtype: fp8
tokens_per_block: 64
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 32768
max_tokens_in_buffer: 32832
backend: UCX
disable_overlap_scheduler: true
speculative_config: &id001
Expand All @@ -85,6 +87,8 @@ worker_config:
print_iter_log: true
max_batch_size: 1
max_num_tokens: 32784
max_input_len: 32784
max_seq_len: 32832
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
Expand All @@ -94,14 +98,14 @@ worker_config:
cuda_graph_config: null
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
free_gpu_memory_fraction: 0.4
dtype: fp8
tokens_per_block: 64
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 32768
max_tokens_in_buffer: 32832
backend: UCX
disable_overlap_scheduler: true
speculative_config: *id001
Loading