From eb0b7279c0a71d40caa4ec40dd2f9ba3f08c8eb0 Mon Sep 17 00:00:00 2001 From: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com> Date: Wed, 17 Jun 2026 13:06:20 -0700 Subject: [PATCH 1/2] [nvbugs/6329052][fix] Drop deepseek_v3_lite_bf16 disagg-workers test from QA cross-GPU list The QA cross-GPU test list (tests/integration/test_lists/qa/llm_function_core.txt) carried test_workers.py::test_workers_conditional_disaggregation_deepseek_v3_lite_bf16, even though the test's only test-db entry is l0_dgx_h100.yml. When QA ran that list against the L40S pool, background_workers() collapsed both ctx and gen workers onto a single L40S (44 GiB), where two ~40 GiB DeepSeek-V3-Lite/bf16 weight copies cannot coexist - second worker OOMs in model_loader.py:init_meta_tensor. Two ~40 GiB copies on a 44 GiB device is a hard hardware limit, not a budgeting bug: weights alone (independent of free_gpu_memory_fraction or max_num_tokens) exceed device capacity. The fix is at the QA-list level: - Remove the test from llm_function_core.txt so the cross-GPU QA pipeline no longer collects it on hardware that cannot satisfy its memory needs. - Remove the now-redundant L40S waiver in waives.txt. The DGX-H100 CI coverage is unchanged - the test remains in test_lists/test-db/l0_dgx_h100.yml. Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com> --- tests/integration/test_lists/qa/llm_function_core.txt | 1 - tests/integration/test_lists/waives.txt | 2 -- 2 files changed, 3 deletions(-) diff --git a/tests/integration/test_lists/qa/llm_function_core.txt b/tests/integration/test_lists/qa/llm_function_core.txt index a9254c066cea..07073c0a8c7b 100644 --- a/tests/integration/test_lists/qa/llm_function_core.txt +++ b/tests/integration/test_lists/qa/llm_function_core.txt @@ -924,7 +924,6 @@ disaggregated/test_disaggregated_single_gpu.py::test_disaggregated_simple_qwen3[ disaggregated/test_disaggregated_single_gpu.py::test_disaggregated_spec_dec_batch_slot_limit[False-False-EAGLE3-LLaMA3.1-Instruct-8B-Llama-3.1-8B-Instruct] disaggregated/test_disaggregated_single_gpu.py::test_disaggregated_spec_dec_batch_slot_limit[True-False-EAGLE3-LLaMA3.1-Instruct-8B-Llama-3.1-8B-Instruct] disaggregated/test_workers.py::test_workers_conditional_disaggregation[TinyLlama-1.1B-Chat-v1.0] -disaggregated/test_workers.py::test_workers_conditional_disaggregation_deepseek_v3_lite_bf16[DeepSeek-V3-Lite-bf16] disaggregated/test_workers.py::test_workers_kv_cache_aware_router[TinyLlama-1.1B-Chat-v1.0] disaggregated/test_workers.py::test_workers_kv_cache_aware_router_deepseek_v3_lite_bf16[DeepSeek-V3-Lite-bf16] disaggregated/test_workers.py::test_workers_kv_cache_aware_router_eviction[TinyLlama-1.1B-Chat-v1.0] diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index 54ee277060a7..a2a8b482d9a2 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -146,7 +146,6 @@ examples/visual_gen/test_visual_gen_multi_gpu.py::test_wan22_t2v_lpips_against_g examples/visual_gen/test_visual_gen_multi_gpu.py::test_wan22_t2v_lpips_against_golden_tp[cfg2_tp2] SKIP (https://nvbugs/6329227) examples/visual_gen/test_visual_gen_multi_gpu.py::test_wan22_t2v_lpips_against_golden_tp[tp2] SKIP (https://nvbugs/6329227) examples/visual_gen/test_visual_gen_multi_gpu.py::test_wan22_t2v_lpips_against_golden_tp[tp2_ulysses2] SKIP (https://nvbugs/6329227) -full:A100/disaggregated/test_workers.py::test_workers_conditional_disaggregation_deepseek_v3_lite_bf16[DeepSeek-V3-Lite-bf16] SKIP (https://nvbugs/6329052) full:A100X/llmapi/test_llm_examples.py::test_llmapi_speculative_decoding_mtp SKIP (https://nvbugs/6287561) full:A100X/unittest/llmapi/test_llm_pytorch.py -m "part0" SKIP (https://nvbugs/6416249) full:B200/accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[throughput_pp4_mtp] SKIP (https://nvbugs/5970614) @@ -219,7 +218,6 @@ full:L40S/accuracy/test_llm_api_autodeploy.py::TestNemotronH::test_auto_dtype[tr full:L40S/accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_fp8[latency-torch_compile=True] SKIP (https://nvbugs/6276841) full:L40S/accuracy/test_llm_api_pytorch_multimodal.py::TestExaone4_5_33B::test_auto_dtype[forced_chunked_prefill] SKIP (https://nvbugs/6327149) full:L40S/accuracy/test_llm_api_pytorch_multimodal.py::TestExaone4_5_33B::test_auto_dtype[full_budget] SKIP (https://nvbugs/6327149) -full:L40S/disaggregated/test_workers.py::test_workers_conditional_disaggregation_deepseek_v3_lite_bf16[DeepSeek-V3-Lite-bf16] SKIP (https://nvbugs/6329052) full:RTXPro6000D/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-ep4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=False] SKIP (https://nvbugs/5948435) full:RTXPro6000D/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-ep4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=True] SKIP (https://nvbugs/5961814) full:RTXPro6000D/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=2-ep4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=False] SKIP (https://nvbugs/5961814) From 62e70c66c12417787562d25a7d035ca462eb4cf0 Mon Sep 17 00:00:00 2001 From: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com> Date: Wed, 17 Jun 2026 14:00:41 -0700 Subject: [PATCH 2/2] [nvbugs/6329052][fix] Use FLASHINFER + 4 layers for DeepSeek-V3-Lite disagg conditional test Run the workers conditional-disaggregation test for DeepSeek-V3-Lite/bf16 with attn_backend=FLASHINFER and num_hidden_layers=4 so it can pass on a single 44 GiB L40S host (and runs faster on multi-GPU hosts). Two ~38 GiB worker copies of the full 30-layer bf16 checkpoint cannot share a 44 GiB GPU (hard hardware limit; weights alone exceed device capacity, see the OOM at model_loader.py:468 init_meta_tensor). Reducing to 4 layers shrinks per-worker weight footprint by ~7x so two workers fit. The default TRTLLM attn backend asserts in attentionOp.cpp:3091 'Deepseek should be supported by fmha in generation part.' on SM89; FLASHINFER provides an MLA path that does not depend on the SM90 FMHA cubin set. The test exercises disagg orchestration (router decisions, KV cache events, prefix matching, multi-round chat) -- not model accuracy -- so the smaller layer count and alternative attention backend do not change what is being verified. The YAML is consumed only by this test. Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com> --- .../test_configs/disagg_config_cache_reuse_deepseek_v3.yaml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/integration/defs/disaggregated/test_configs/disagg_config_cache_reuse_deepseek_v3.yaml b/tests/integration/defs/disaggregated/test_configs/disagg_config_cache_reuse_deepseek_v3.yaml index 36b459dabc1e..9a51a0f5903d 100644 --- a/tests/integration/defs/disaggregated/test_configs/disagg_config_cache_reuse_deepseek_v3.yaml +++ b/tests/integration/defs/disaggregated/test_configs/disagg_config_cache_reuse_deepseek_v3.yaml @@ -2,9 +2,12 @@ hostname: localhost model: DeepSeek-V3-Lite/bf16 free_gpu_memory_fraction: 0.15 backend: pytorch +attn_backend: FLASHINFER cuda_graph_config: null disable_overlap_scheduler: true enable_autotuner: false +model_kwargs: + num_hidden_layers: 4 context_servers: num_instances: 1 tensor_parallel_size: 1