Skip to content

[https://nvbugs/5844149][fix] Fix issues with DSV3.2 perf tests #13142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chenfeiz0326 merged 5 commits into from
Apr 17, 2026
Merged

[https://nvbugs/5844149][fix] Fix issues with DSV3.2 perf tests #13142

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b82c6c6
[https://nvbugs/5844149][fix] Fix OOM in ctx server and IMA in gen se…
brb-nv Apr 9, 2026
e0838e9
cleanup
brb-nv Apr 14, 2026
d0176fe
Move multi round = 1 to test_perf_sanity.py
chenfeiz0326 Apr 17, 2026
8f9bba1
Move multi round = 1 to test_perf_sanity.py
chenfeiz0326 Apr 17, 2026
36b41f2
update
chenfeiz0326 Apr 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion tests/integration/defs/perf/test_perf_sanity.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1391,7 +1391,9 @@ def _parse_disagg_config_file(self, config_file_path: str, config_file: str):
for concurrency in concurrency_values:
client_config_data = {
"concurrency": concurrency,
"iterations": benchmark.get("multi_round", 1),
"iterations": 1
if benchmark_mode == "gen_only"
else benchmark.get("multi_round", 1),
"isl": benchmark.get("input_length", 1024),
"osl": osl,
"random_range_ratio": benchmark.get("benchmark_ratio", 0.0),
Expand Down
4 changes: 1 addition & 3 deletions tests/integration/test_lists/waives.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,9 +325,7 @@ disaggregated/test_disaggregated.py::test_disaggregated_gpt_oss_120b_harmony[gpt
accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[pp4-mtp_nextn=0-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=False] SKIP (https://nvbugs/6050489)
disaggregated/test_disaggregated.py::test_disaggregated_overlap_gen_first[ctx_pp1-TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugs/6057459)
disaggregated/test_disaggregated.py::test_disaggregated_overlap_gen_first[ctx_pp4-TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugs/6057460)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_32k4k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX] SKIP (https://nvbugs/5844149)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-UCX] SKIP (https://nvbugs/5844149)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_32k4k_con256_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-UCX] SKIP (https://nvbugs/5844149)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-UCX] SKIP (https://nvbugs/6085022)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-r1-fp4_128k8k_con128_ctx1_pp8_gen1_dep16_eplb0_mtp2_ccb-UCX] SKIP (https://nvbugs/6060119)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-r1-fp4_128k8k_con64_ctx1_pp8_gen1_dep32_eplb0_mtp3_ccb-UCX] SKIP (https://nvbugs/6060119)
full:sm89/accuracy/test_llm_api_pytorch.py::TestGemma3_1BInstruct::test_fp8_prequantized[torch_compile=False] SKIP (https://nvbugs/6070878)
Expand Down
15 changes: 10 additions & 5 deletions ...saggregated/gb200_deepseek-v32-fp4_32k4k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-UCX.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,24 +48,27 @@ worker_config:
gen:
print_iter_log: true
max_batch_size: 1
max_num_tokens: 256
max_num_tokens: 8
max_input_len: 32784
max_seq_len: 40960
tensor_parallel_size: 8
moe_expert_parallel_size: 8
pipeline_parallel_size: 1
context_parallel_size: 1
enable_attention_dp: false
enable_lm_head_tp_in_adp: false
cuda_graph_config:
enable_padding: true
max_batch_size: 1
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.9
free_gpu_memory_fraction: 0.25
dtype: fp8
tokens_per_block: 64
moe_config:
backend: TRTLLM
cache_transceiver_config:
max_tokens_in_buffer: 32768
max_tokens_in_buffer: 32832
backend: UCX
disable_overlap_scheduler: true
speculative_config: &id001
Expand All @@ -84,6 +87,8 @@ worker_config:
print_iter_log: true
max_batch_size: 1
max_num_tokens: 32784
max_input_len: 32784
max_seq_len: 32832
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
Expand All @@ -93,14 +98,14 @@ worker_config:
cuda_graph_config: null
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
free_gpu_memory_fraction: 0.4
dtype: fp8
tokens_per_block: 64
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 32768
max_tokens_in_buffer: 32832
backend: UCX
disable_overlap_scheduler: true
speculative_config: *id001
12 changes: 8 additions & 4 deletions ...gated/gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-UCX.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ worker_config:
print_iter_log: true
max_batch_size: 64
max_num_tokens: 128
max_input_len: 32784
max_seq_len: 40960
tensor_parallel_size: 32
moe_expert_parallel_size: 32
pipeline_parallel_size: 1
Expand All @@ -60,7 +62,7 @@ worker_config:
max_batch_size: 64
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.85
free_gpu_memory_fraction: 0.5
dtype: fp8
tokens_per_block: 64
moe_config:
Expand All @@ -70,7 +72,7 @@ worker_config:
num_slots: 288
layer_updates_per_iter: 1
cache_transceiver_config:
max_tokens_in_buffer: 32768
max_tokens_in_buffer: 32832
backend: UCX
disable_overlap_scheduler: true
speculative_config: &id001
Expand All @@ -88,6 +90,8 @@ worker_config:
print_iter_log: true
max_batch_size: 1
max_num_tokens: 32784
max_input_len: 32784
max_seq_len: 32832
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
Expand All @@ -97,14 +101,14 @@ worker_config:
cuda_graph_config: null
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
free_gpu_memory_fraction: 0.4
dtype: fp8
tokens_per_block: 64
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 32768
max_tokens_in_buffer: 32832
backend: UCX
disable_overlap_scheduler: true
speculative_config: *id001
14 changes: 9 additions & 5 deletions ...gregated/gb200_deepseek-v32-fp4_32k4k_con256_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-UCX.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,9 @@ worker_config:
gen:
print_iter_log: true
max_batch_size: 8
max_num_tokens: 256
max_num_tokens: 64
max_input_len: 32784
max_seq_len: 40960
tensor_parallel_size: 32
moe_expert_parallel_size: 32
pipeline_parallel_size: 1
Expand All @@ -60,14 +62,14 @@ worker_config:
max_batch_size: 8
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.85
free_gpu_memory_fraction: 0.5
dtype: fp8
tokens_per_block: 64
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 32768
max_tokens_in_buffer: 32832
backend: UCX
disable_overlap_scheduler: true
speculative_config: &id001
Expand All @@ -85,6 +87,8 @@ worker_config:
print_iter_log: true
max_batch_size: 1
max_num_tokens: 32784
max_input_len: 32784
max_seq_len: 32832
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
Expand All @@ -94,14 +98,14 @@ worker_config:
cuda_graph_config: null
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.6
free_gpu_memory_fraction: 0.4
dtype: fp8
tokens_per_block: 64
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 32768
max_tokens_in_buffer: 32832
backend: UCX
disable_overlap_scheduler: true
speculative_config: *id001
Loading