Skip to content

[None][test] Add qa disagg test cases for l0 local model for release 1.2 #12070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fredricz-20070104 wants to merge 1 commit into from
Closed

[None][test] Add qa disagg test cases for l0 local model for release 1.2 #12070

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions tests/integration/defs/perf/test_perf_sanity.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@
"deepseek_r1_0528_fp4": "DeepSeek-R1/DeepSeek-R1-0528-FP4/",
"deepseek_r1_0528_fp4_v2": "DeepSeek-R1/DeepSeek-R1-0528-FP4-v2/",
"gpt_oss_120b_fp4": "gpt_oss/gpt-oss-120b",
"k2_thinking_fp4": "Kimi-K2-Thinking-NVFP4",
"qwen3_235b_a22b_fp4": "Qwen3/saved_models_Qwen3-235B-A22B_nvfp4_hf", # Qwen3-235B-A22B-FP4
"qwen3_235b_a22b_fp8": "Qwen3/saved_models_Qwen3-235B-A22B_fp8_hf", # Qwen3-235B-A22B-FP8
Comment on lines +58 to +60

@coderabbitai coderabbitai Bot Mar 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add the deepseek_v32_fp4 model-path mapping too.

The new wideep_deepseek-v32-fp4_* configs in this PR use model_name: deepseek_v32_fp4, but get_model_dir() only resolves keys present in MODEL_PATH_DICT. Without an entry here, the runner falls back to the literal string deepseek_v32_fp4 instead of the synced local model directory.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@tests/integration/defs/perf/test_perf_sanity.py` around lines 58 - 60, Add a
mapping for the new model key so get_model_dir() can resolve it: update
MODEL_PATH_DICT to include "deepseek_v32_fp4" mapped to the synced local model
path, e.g. add the entry "deepseek_v32_fp4":
"Wideep/saved_models_wideep_deepseek-v32-fp4_hf" (follow the same naming
convention as the other entries) so get_model_dir() returns the local directory
instead of the literal key.

}

SUPPORTED_GPU_TYPE = [
Expand Down
162 changes: 162 additions & 0 deletions tests/integration/test_lists/qa/llm_perf_multinode.txt
View file
Open in desktop

Large diffs are not rendered by default.

106 changes: 106 additions & 0 deletions ...aggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-NIXL.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
metadata:
model_name: qwen3_235b_a22b_fp4
precision: fp4
model_dir_name: Qwen3-235B-A22B-FP4
supported_gpus:
- GB200
- GB300
script_file: disaggr_torch.slurm
benchmark_type: 1k1k
slurm:
script_file: disaggr_torch.slurm
partition: <partition>
account: <account>
job_time: 02:00:00
job_name: unified-benchmark
extra_args: --gres=gpu:4
numa_bind: true
benchmark:
mode: e2e
use_nv_sa_benchmark: true
multi_round: 8
benchmark_ratio: 0.8
streaming: true
concurrency_list: '1024'
input_length: 1024
output_length: 1024
dataset_file: <dataset_file>
hardware:
gpus_per_node: 4
num_ctx_servers: 1
num_gen_servers: 1
environment:
container_mount: <container_mount>
container_image: <container_image>
model_path: <model_path>
trtllm_repo: ''
build_wheel: false
work_dir: <full_path_to_work_dir>
worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1
TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes
server_env_var: TRTLLM_SERVER_DISABLE_GC=1
profiling:
nsys_on: false
accuracy:
enable_accuracy_test: false
worker_config:
gen:
tensor_parallel_size: 16
moe_expert_parallel_size: 16
enable_attention_dp: true
pipeline_parallel_size: 1
max_batch_size: 64
max_num_tokens: 256
max_seq_len: 2251
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
- 16
- 32
- 64
- 128
- 256
- 512
- 768
- 1024
- 2048
print_iter_log: true
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.7
dtype: fp8
moe_config:
backend: WIDEEP
cache_transceiver_config:
max_tokens_in_buffer: 4608
backend: NIXL
stream_interval: 20
num_postprocess_workers: 4
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
ctx:
max_batch_size: 4
max_num_tokens: 4608
max_seq_len: 2251
tensor_parallel_size: 4
moe_expert_parallel_size: 4
enable_attention_dp: true
pipeline_parallel_size: 1
print_iter_log: true
cuda_graph_config: null
disable_overlap_scheduler: true
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.85
dtype: fp8
cache_transceiver_config:
max_tokens_in_buffer: 4608
backend: NIXL
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
Comment on lines +86 to +106

@coderabbitai coderabbitai Bot Mar 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Pin the ctx MoE backend explicitly.

gen.moe_config.backend is fixed to WIDEEP, but ctx relies on the default. That makes this QA case sensitive to default-backend changes and can benchmark prefill/decode with different MoE implementations.

🔧 Suggested patch 
   ctx:
     max_batch_size: 4
     max_num_tokens: 4608
     max_seq_len: 2251
     tensor_parallel_size: 4
     moe_expert_parallel_size: 4
     enable_attention_dp: true
     pipeline_parallel_size: 1
     print_iter_log: true
     cuda_graph_config: null
     disable_overlap_scheduler: true
     kv_cache_config:
       enable_block_reuse: false
       free_gpu_memory_fraction: 0.85
       dtype: fp8
     cache_transceiver_config:
       max_tokens_in_buffer: 4608
       backend: NIXL
+    moe_config:
+      backend: <intended_ctx_moe_backend>
     speculative_config:
       decoding_type: MTP
       num_nextn_predict_layers: 3
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In
`@tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-NIXL.yaml`
around lines 86 - 106, The test currently leaves ctx's MoE backend to defaults
while gen.moe_config.backend is set to WIDEEP; make the case deterministic by
explicitly setting ctx.moe_config.backend to the same backend (e.g., WIDEEP) in
the YAML ctx block so prefill/decode use the intended MoE implementation; locate
the ctx section and add a moe_config key with backend: WIDEEP (or mirror
gen.moe_config.backend) to ensure the benchmark is not sensitive to
default-backend changes.

106 changes: 106 additions & 0 deletions ...saggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-UCX.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
metadata:
model_name: qwen3_235b_a22b_fp4
precision: fp4
model_dir_name: Qwen3-235B-A22B-FP4
supported_gpus:
- GB200
- GB300
script_file: disaggr_torch.slurm
benchmark_type: 1k1k
slurm:
script_file: disaggr_torch.slurm
partition: <partition>
account: <account>
job_time: 02:00:00
job_name: unified-benchmark
extra_args: --gres=gpu:4
numa_bind: true
benchmark:
mode: e2e
use_nv_sa_benchmark: true
multi_round: 8
benchmark_ratio: 0.8
streaming: true
concurrency_list: '1024'
input_length: 1024
output_length: 1024
dataset_file: <dataset_file>
hardware:
gpus_per_node: 4
num_ctx_servers: 1
num_gen_servers: 1
environment:
container_mount: <container_mount>
container_image: <container_image>
model_path: <model_path>
trtllm_repo: ''
build_wheel: false
work_dir: <full_path_to_work_dir>
worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1
TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes
server_env_var: TRTLLM_SERVER_DISABLE_GC=1
profiling:
nsys_on: false
accuracy:
enable_accuracy_test: false
worker_config:
gen:
tensor_parallel_size: 16
moe_expert_parallel_size: 16
enable_attention_dp: true
pipeline_parallel_size: 1
max_batch_size: 64
max_num_tokens: 256
max_seq_len: 2251
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
- 16
- 32
- 64
- 128
- 256
- 512
- 768
- 1024
- 2048
print_iter_log: true
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.7
dtype: fp8
moe_config:
backend: WIDEEP
cache_transceiver_config:
max_tokens_in_buffer: 4608
backend: UCX
stream_interval: 20
num_postprocess_workers: 4
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
ctx:
max_batch_size: 4
max_num_tokens: 4608
max_seq_len: 2251
tensor_parallel_size: 4
moe_expert_parallel_size: 4
enable_attention_dp: true
pipeline_parallel_size: 1
print_iter_log: true
cuda_graph_config: null
disable_overlap_scheduler: true
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.85
dtype: fp8
cache_transceiver_config:
max_tokens_in_buffer: 4608
backend: UCX
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
106 changes: 106 additions & 0 deletions ...saggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con512_ccb-NIXL.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
metadata:
model_name: qwen3_235b_a22b_fp4
precision: fp4
model_dir_name: Qwen3-235B-A22B-FP4
supported_gpus:
- GB200
- GB300
script_file: disaggr_torch.slurm
benchmark_type: 1k1k
slurm:
script_file: disaggr_torch.slurm
partition: <partition>
account: <account>
job_time: 02:00:00
job_name: unified-benchmark
extra_args: --gres=gpu:4
numa_bind: true
benchmark:
mode: e2e
use_nv_sa_benchmark: true
multi_round: 8
benchmark_ratio: 0.8
streaming: true
concurrency_list: '512'
input_length: 1024
output_length: 1024
dataset_file: <dataset_file>
hardware:
gpus_per_node: 4
num_ctx_servers: 1
num_gen_servers: 1
environment:
container_mount: <container_mount>
container_image: <container_image>
model_path: <model_path>
trtllm_repo: ''
build_wheel: false
work_dir: <full_path_to_work_dir>
worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1
TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes
server_env_var: TRTLLM_SERVER_DISABLE_GC=1
profiling:
nsys_on: false
accuracy:
enable_accuracy_test: false
worker_config:
gen:
tensor_parallel_size: 16
moe_expert_parallel_size: 16
enable_attention_dp: true
pipeline_parallel_size: 1
max_batch_size: 64
max_num_tokens: 256
max_seq_len: 2251
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
- 16
- 32
- 64
- 128
- 256
- 512
- 768
- 1024
- 2048
print_iter_log: true
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.7
dtype: fp8
moe_config:
backend: WIDEEP
cache_transceiver_config:
max_tokens_in_buffer: 4608
backend: NIXL
stream_interval: 20
num_postprocess_workers: 4
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
ctx:
max_batch_size: 4
max_num_tokens: 4608
max_seq_len: 2251
tensor_parallel_size: 4
moe_expert_parallel_size: 4
enable_attention_dp: true
pipeline_parallel_size: 1
print_iter_log: true
cuda_graph_config: null
disable_overlap_scheduler: true
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.85
dtype: fp8
cache_transceiver_config:
max_tokens_in_buffer: 4608
backend: NIXL
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
Loading