[trtllm-agent]

Description

Temporarily waive 12 failed case(s) on DGX_H100. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_batch_cache_indices_honor_requested_blocks broken on main for DGX_H100: failure signature tensorrt_llm.bindings.internal.batch_manager.kv_ca....hostoomerror: mmap failed: cannot allocate memory, onset 2026-11-05T08:25:22Z, latest failure 2026-11-05T19:23:27Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7053307 tracks the fix. Scope DGX_H100 is the stage prefix that ran the failing test.

• Bug: https://nvbugs/7053307 (details, logs and 7-day test history are in the bug)
• Restore coverage when: Remove this line when https://nvbugs/7053307 is fixed or closed and the test passes on DGX_H100 post-merge; un-waive follows the bug, not the PR.
• Scope reviewed by: auto-waive at 2026-11-05T21:37:55.880696+00:00

Tests in this break event:

• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_batch_cache_indices_honor_requested_blocks on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_batch_cache_indices_skip_padded_entries on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_per_layer_head_dim_with_equal_buffer_sizes on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_uniform_head_dim_int on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_uniform_head_dim_list on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimHeterogeneous::test_per_layer_head_dim_heterogeneous on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[False-DataType.FP8] on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[False-DataType.HALF] on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[False-DataType.NVFP4] on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[True-DataType.FP8] on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[True-DataType.HALF] on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[True-DataType.NVFP4] on DGX_H100

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

• [TRTLLMINF-451][infra] Capture setup timeouts and defer unfinished cleanup until JUnit save #19333
• [https://nvbugs/6858496][fix] Fail disagg gen requests whose grammar matcher cannot be created #20903
• [TRTLLM-15944][feat] Enable SWA scratch reuse for Gemma3/4 with TRTLLM attention #20971
• [https://nvbugs/6891165][fix] Pass chat_template_kwargs in the request body of the JSON-schema serve example #21026
• [None][feat] Rubin merge back: MLA, DSpark, DWDP, Helix #21067
• [None][feat] Keep hybrid speculative replay state in private KV cache buffers #21324
• [None][feat] GEN slot preemption: pause decode requests in place to admit at-risk waiters #21334
• [TRTLLM-17272][feat] Add public LLM health queries and a private-access lint for the serving frontends #21393
• [None][fix] Fix draft KV over-allocation during chunked prefill #21546
• [TRTLLM-16624][fix] Fix Helix MLA decode kernel selection #21582
• [TRTLLM-16731][fix] Autotuner: skip redundant per-profile work for exclude_from_cache ops in tuning mode #21593
• [None][feat] Responses: name the engine request behind each reply #21602
• [None][perf] Build MSA fmha_sm100 JIT kernels for the visible GPUs' target only #21681

full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_batch_cache_indices_honor_requested_blocks SKIP (https://nvbugs/7053307)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_batch_cache_indices_skip_padded_entries SKIP (https://nvbugs/7053307)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_per_layer_head_dim_with_equal_buffer_sizes SKIP (https://nvbugs/7053307)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_uniform_head_dim_int SKIP (https://nvbugs/7053307)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_uniform_head_dim_list SKIP (https://nvbugs/7053307)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimHeterogeneous::test_per_layer_head_dim_heterogeneous SKIP (https://nvbugs/7053307)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[False-DataType.FP8] SKIP (https://nvbugs/7053307)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[False-DataType.HALF] SKIP (https://nvbugs/7053307)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[False-DataType.NVFP4] SKIP (https://nvbugs/7053307)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[True-DataType.FP8] SKIP (https://nvbugs/7053307)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[True-DataType.HALF] SKIP (https://nvbugs/7053307)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[True-DataType.NVFP4] SKIP (https://nvbugs/7053307)

What to check before merging:

• The bug's test-history links show the cases failing on main, not only on the trigger PR.
• The scope DGX_H100 matches the failing stage and no narrower waiver already exists.
• Nothing else is needed from you after merging: the bug owner removes these lines with the fix.

Provenance

• Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
• Run 91685f6e-5aff-47e8-b513-9a331b095319; snapshot SHA-256 6b9de299e0a60b16f56c9f5e3f10cc7bc240b9439bec64e644b42f4db79d9f9f; config SHA-256 622a43f13fb241c37a015e4ec2c812d7c8cecc1a04413b2618ca4ac119a77c81
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_batch_cache_indices_honor_requested_blocks: incident 0296cf6f0bcbb1e7898937f99424e26efa68b385dae6cbef277b06be66812345, observation 08dc018d511ff434b131df04f139bdb0603bf9f83b970ac54000496f72d0b7f6
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_batch_cache_indices_skip_padded_entries: incident d94d92eb1dd7cf8b4b37f9f381703640917845129114a1da73668844736fb395, observation 73dd1bb1f88f091ab3719287417b4bcdd1d78f7bbcca7195b61212f56ab53bb1
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_per_layer_head_dim_with_equal_buffer_sizes: incident 04f6023d27a54e545ada1552e1d659119e6c7349e120ea8ca40e086c03ff5f33, observation b2df794d06c30b3239ca9a13fce8e88c6aee1695c02961f11d78e7e8a7b0679f
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_uniform_head_dim_int: incident d1896b63636fc9b1dd847eaca00632ee868920c60a79650c791ad1514ae6cc22, observation d2802c943bb9a41a6cb6ee01949b52cd5678920ba2d6bef2ffbc7948d26c4f0a
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimBasic::test_uniform_head_dim_list: incident 4804872b9ccadbd9ced614fed730c70c0c5759fdbb0ca25cbfb051a56f6745a2, observation 594d0e3de4b3ebc03f26fc88eb015ff7c31de5efb9e877cc11de3a7c94dbd6d5
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::TestPerLayerHeadDimHeterogeneous::test_per_layer_head_dim_heterogeneous: incident 1594381338c29597c7f1d5b78c31efe3be49c5cef527a477267dd7db658dd9d3, observation 68336b9296ec258fd029ba4b5e440179871226767ebb1a103b8c2d717f763099
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[False-DataType.FP8]: incident 8b60ced560f3bf22f6cd5729f0fab4fb038e668daca5c7d895b479734be502f4, observation 3bc352a174ef2a0175d9ee0c560bb483cb9aa0ac2da351beecd4c26ef468c599
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[False-DataType.HALF]: incident f4bf3e47e5b1843aac7fa0918001700a7a594d5c77c5fa618acd4333f6f72941, observation 43c67d6244f89a511b3c4dfa636392437c39cd7f1eb9e68778482fc4755e7d44
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[False-DataType.NVFP4]: incident bf3e73461a34c8fd9d34c1b06593e3e14c6f15c789868b2c08dbd005e3fa5dfe, observation f3ed10578a83f9510449b4ec2d05348389169b82e3bd2fca6630ada883e19015
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[True-DataType.FP8]: incident 5ea3c1465338b926a9770615325f32b4d21d0d4b027821fb63648a1447d178fa, observation 2e854d7d7f710f778b51d88dd8b45b4b3888e6e625a28a69cb6e363459032857
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[True-DataType.HALF]: incident 0a903577579e947ffb99ef0d397f7ecc89b189aa68fed37a8a83941608deb59d, observation 5d9c5ec7230f24cf4d7476b4e0c00beab0218d3d949c955d47c2b65a46565e49
• unittest/_torch/pyexecutor/kv_cache/test_per_layer_head_dim.py::test_heterogeneous_page_tables_match_allocated_addresses[True-DataType.NVFP4]: incident 1f770e86b10637c524c929b0fa2512e772eb7d426beaa7245704b9623e0b8555, observation f85e5233f6114637764b35efbe57e00c793cba6d15db263a72850bc83efdc380

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

• Please check this after reviewing the above items as appropriate for this PR.

[karljang]

Closing: these failures were a DGX_H100 host-memory condition (host OOM / CUDA invalid device context across 16 nodes, 37 of 50 builds passing), not a code break, and the stage recovered on its own. No waiver needed. The linked NVBug is being closed as not a bug.