Skip to content

[https://nvbugs/7050981][infra] Waive 7 failed case(s) on DGX_B200 - #21721

Draft
trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/cbae1531f402c77cc6a64ed6bc3e6d09
Draft

trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/cbae1531f402c77cc6a64ed6bc3e6d09

Conversation

@trtllm-agent

Copy link
Copy Markdown
Collaborator

Description

Temporarily waive 7 failed case(s) on DGX_B200. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_chunked_prefill[quant_dtype=nvfp4-kv_cache_reuse=True-fp8kv=False-overlap_scheduler=True] broken on main for DGX_B200: failure signature test terminated unexpectedly, onset 2026-11-05T08:08:49.255000Z, latest failure 2026-11-05T08:19:34.838000Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7050981 tracks the fix. Scope DGX_B200 is the stage prefix that ran the failing test.

  • Bug: https://nvbugs/7050981 (details, logs and 7-day test history are in the bug)
  • Restore coverage when: Remove this line when https://nvbugs/7050981 is fixed or closed and the test passes on DGX_B200 post-merge; un-waive follows the bug, not the PR.
  • Scope reviewed by: auto-waive at 2026-11-05T12:05:48.558424+00:00

Tests in this break event:

  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_chunked_prefill[quant_dtype=nvfp4-kv_cache_reuse=True-fp8kv=False-overlap_scheduler=True] on DGX_B200
  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales[mtp=disable-fp8kv=False-attention_dp=False-cuda_graph=True-overlap_scheduler=False-torch_compile=False] on DGX_B200
  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4[moe_backend=CUTLASS-mtp_nextn=0-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=True] on DGX_B200
  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_prims_ts_bfloat16 on DGX_B200
  • accuracy/test_llm_api_pytorch.py::TestQwen3_5_35B_A3B::test_fp8[enable_block_reuse=True] on DGX_B200
  • accuracy/test_llm_api_pytorch.py::TestQwen3_6_35B_A3B::test_nvfp4_dflash[TRTLLM] on DGX_B200
  • accuracy/test_llm_api_pytorch.py::TestQwen3_6_35B_A3B::test_nvfp4_dflash[VANILLA] on DGX_B200

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

full:DGX_B200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_chunked_prefill[quant_dtype=nvfp4-kv_cache_reuse=True-fp8kv=False-overlap_scheduler=True] SKIP (https://nvbugs/7050981)
full:DGX_B200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales[mtp=disable-fp8kv=False-attention_dp=False-cuda_graph=True-overlap_scheduler=False-torch_compile=False] SKIP (https://nvbugs/7050981)
full:DGX_B200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4[moe_backend=CUTLASS-mtp_nextn=0-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=True] SKIP (https://nvbugs/7050981)
full:DGX_B200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_prims_ts_bfloat16 SKIP (https://nvbugs/7050981)
full:DGX_B200/accuracy/test_llm_api_pytorch.py::TestQwen3_5_35B_A3B::test_fp8[enable_block_reuse=True] SKIP (https://nvbugs/7050981)
full:DGX_B200/accuracy/test_llm_api_pytorch.py::TestQwen3_6_35B_A3B::test_nvfp4_dflash[TRTLLM] SKIP (https://nvbugs/7050981)
full:DGX_B200/accuracy/test_llm_api_pytorch.py::TestQwen3_6_35B_A3B::test_nvfp4_dflash[VANILLA] SKIP (https://nvbugs/7050981)

What to check before merging:

  • The bug's test-history links show the cases failing on main, not only on the trigger PR.
  • The scope DGX_B200 matches the failing stage and no narrower waiver already exists.
  • Nothing else is needed from you after merging: the bug owner removes these lines with the fix.
Provenance
  • Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
  • Run e2720be2-2960-4b4f-8b4d-e44a5b4d75ca; snapshot SHA-256 19a4487913c2b0de0cd050147e2bb9cf651108766e2ab3ed423f9a0786744687; config SHA-256 09cf0ee078a35cefbb2080f3d321976a2a18bc5d6e6c6b3de9d637db1c92730b
  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_chunked_prefill[quant_dtype=nvfp4-kv_cache_reuse=True-fp8kv=False-overlap_scheduler=True]: incident c8c62fd66327624608cf86578cfa54136869386331c68b63ec3871dbc5359639, observation 7098195a22b0a9bb1d12f2e5df9d396b65ccd2735d04e9cee53d76d3a2ca88e7
  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales[mtp=disable-fp8kv=False-attention_dp=False-cuda_graph=True-overlap_scheduler=False-torch_compile=False]: incident 4aa0648a39ee079dc13b64bad6132d7eed2bcec940aae912eee9c4a991cb6955, observation d23d2f8e8de0e626c5256aaf9b5b15cb579772861964415deb3179c44d3615c3
  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4[moe_backend=CUTLASS-mtp_nextn=0-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=True]: incident f84141ee5e21d409722b2ac8ff5342d225484c42d5f549c20e8606983dcc2a90, observation 477f5bf26a0eca8c38a0d37464ad34f0a5ca0b35095650398d6d34c210a0c7ce
  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_prims_ts_bfloat16: incident c259b45aa6eb07b9035cea1579ae92d25a20af949c818caca83050156bc5b71c, observation df248fad60c13ea0338ecaa6b35645de6f2aa3cceead022cea9179b4e8ac62ae
  • accuracy/test_llm_api_pytorch.py::TestQwen3_5_35B_A3B::test_fp8[enable_block_reuse=True]: incident 1e45604024a51aaa3a25c66cfed78550e0098d9e6a7a9789cfdd988f76bdd382, observation 93d50b95023e0d1aaaf2454639103af56b0c0e8f642a1e2c72396c15619bed28
  • accuracy/test_llm_api_pytorch.py::TestQwen3_6_35B_A3B::test_nvfp4_dflash[TRTLLM]: incident a183c0fb2f6521890cdd9623c3033753f392d15cb98c8220ff8e769a7de3e9e8, observation 9147b955d82e143cd93c2a99ed722a19d9a789cfc55a4aaea27ad699bfd7bbe8
  • accuracy/test_llm_api_pytorch.py::TestQwen3_6_35B_A3B::test_nvfp4_dflash[VANILLA]: incident 7d8b666cec01e8ae9a949d75b8476b2031ca6b7bb384fbd04978854d145cfe1f, observation fcebec22f64b555606b5838b391597f26fd0e24aa70de42fe4a719cbb8fbff5b

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant