[https://nvbugs/6682113][fix] give gb300 v4-pro con4301 ctx_only perf-sanity the 120min budget its workload already carries

[chenfeiz0326]

Description

perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_deepseek-v4-pro-fp4_8k1k_con4301_ctx12_dep4_gen1_dep8_eplb384_mtp1_ccb-NIXL]
is waived under nvbugs/6682113 because it exceeds its 90-minute CI budget. It is
not failing: it completes all 43,010 requests successfully, every rep. It is
simply given 20% less wall clock than the same workload gets elsewhere in the
same test family.

This PR raises that one row's budget from TIMEOUT (90) to TIMEOUT (120) and
removes its waiver.

Why 120 is the right number, and not a new precedent

The identical workload already runs with a 120-minute budget in its two sibling
rows. All three con4301 rows resolve to the same config file
(tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-fp4_8k1k_con4301_ctx12_dep4_gen1_dep8_eplb384_mtp1_ccb-NIXL.yaml,
concurrency_list: '4301', multi_round: 10 → 43,010 requests):

Row	Test-db	Budget	Status
disagg_upload-e2e-...con4301...	l0_gb300_multi_nodes_perf_sanity_ctx12_node1_gpu4_gen1_node2_gpu8.yml:19	TIMEOUT (120)	passing
disagg_upload-gen_only-...con4301...	l0_gb300_multi_nodes_perf_sanity_ctx12_node1_gpu4_gen1_node2_gpu8.yml:18	TIMEOUT (120)	passing
aggr_upload-ctx_only-...con4301...	l0_gb300_multi_gpus_perf_sanity.yml:28	TIMEOUT (90)	waived

So this change aligns the outlier with the budget its own workload already
carries, rather than inventing a looser one.

Evidence: the case passes, and the wall time is legitimate

Re-run on real GB300 hardware (aws_cmh), 3 reps:

Rep	Slurm job	Benchmark duration	pytest total	vs 5400 s	sacct
1	3594174	4894.03 s	5975.01 s	+575.01	COMPLETED 0:0
2	3594166	4856.82 s	5946.75 s	+546.75	COMPLETED 0:0
3	3594287	4811.41 s	5767.07 s	+367.07	COMPLETED 0:0

Every rep: 1 passed, 43010/43010 successful requests, 0 failed, weight load
2327/2327, no qualified OOM signal, all Slurm steps COMPLETED. The case is
functionally healthy; it misses only the clock.

The benchmark window is fully accounted for, so there is no hidden inefficiency
to fix instead:

• 43010 requests / 8.79 req/s = 4893.1 s vs measured 4894.03 s.
• Summed host_step_time over the run's 10,759 iterations is 4914.6 s against a
  4894 s window — ~99.8% of the window is iteration execution, with no idle
  gap, bubble, or stall.
• Per-iteration cost is flat to 0.07% across the whole run (deciles 2-10 span
  454.13-454.46 ms; p99/p1 = 1.011). Only 3 of 10,759 iterations exceed 600 ms
  and 2 of those are warmup. Cost is exactly linear in request count — there is no
  scaling defect.
• The MoE load balancer is static for this case (layer_updates_per_iter: 0,
  num_slots: 384), so there is no per-iteration expert-map work to remove.
• Prefill throughput is 64,971 tok/s, versus 44,631 tok/s for the con8 ctx_only
  row on the same dep4 shape — this case is 46% faster per token, it is just
  23.9x larger.

Why the budget cannot be met by trimming startup

Startup is 1075 s, measured (rep 1):

Phase	Cost
imports / init	44 s
weight load	530 s (49%)
post-load + KV cache setup	62 s
TRTLLM-Gen FMHA JIT warmup	41 s
run warmup	22 s
autotuner warmup	325 s (30%)
DG warmup + memory profiling	4 s
server ready → first iteration	47 s

Only the autotuner (325 s) and FMHA JIT (41 s) are plausibly reducible, totalling
366 s. Subtracting both entirely — which is neither achievable nor desirable,
since it would corrupt the measurement — still leaves:

• rep 1: 5975 − 366 = 5609 s → +209 s over 5400 s
• rep 2: 5947 − 366 = 5581 s → +181 s over
• rep 3: 5767 − 366 = 5401 s → +1 s over

3/3 reps miss the 90-minute budget even under that maximally generous assumption.
The 530 s weight load is irreducible I/O on an 805 GiB checkpoint. test_perf_sanity.py
already documents this case's exceptional startup in the readiness-timeout comment
("gb300 DeepSeek-V4-Pro ctx_only (con4301) needs ~2000s of model load + autotune
before /health answers", nvbugs/6517846).

At 99.6 minutes worst-case, a 120-minute budget leaves ~20% margin (5975 s vs
7200 s).

Scope

• Touches exactly one test-db row and one waiver line. No source changes.
• Does not alter the workload, so historical metric comparability for this case is
  preserved.
• GB300 only. The waive line carried no full:<PLATFORM>/ prefix and the id is
  declared in exactly one test-db, so the deletion re-enables the case on GB300
  alone.
• The other waived perf-sanity entries are deliberately untouched.

Known follow-up, not addressed here

The 90-vs-120 split appears to be assigned per test-db file rather than per
workload. All three aggr_upload-ctx_only rows in
l0_gb300_multi_gpus_perf_sanity.yml carry TIMEOUT (90) despite spanning a
~24x workload range (con180 → 1,800 requests, con666 → 6,660, con4301 → 43,010),
while every disagg row in the same family carries 120. Only the con4301 row
currently overruns, so this PR fixes just that row. The broader budget-assignment
question belongs with nvbugs/6682328 / 6683835 and is left to their owner.

Note the con180 / con666 figures above are predictions from this case's measured
constants (113.79 ms/request, 1075 s startup), not measurements — no artifacts
were collected for those two rows.

PR Checklist

• Commits are DCO signed off (git commit -s)
• One concern per PR — a single under-budgeted row and its waiver
• Validated on real hardware, 3/3 reps, with the wall time fully accounted for
• Budget matches what the identical workload already carries in its sibling rows

Dev Engineer Review

• Updated the GB300 aggr_upload-ctx_only con4301 perf-sanity entry from TIMEOUT (90) to TIMEOUT (120).
• Removed the matching waiver from waives.txt.
• The values and scope are consistent with sibling configurations.
• No source code, workload, or API changes are included.

QA Engineer Review

• Modified tests/integration/test_lists/test-db/l0_gb300_multi_gpus_perf_sanity.yml by increasing the test timeout.
• Modified tests/integration/test_lists/waives.txt by removing the matching waiver.
• The test identifier and list formatting are valid. The change does not add duplicates or require a bug reference.
• Verdict: sufficient

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 697c4787-6091-4b8f-890a-a7e2df15b0b7

📥 Commits

Reviewing files that changed from the base of the PR and between 3811a81 and 3013909.

📒 Files selected for processing (2)

• tests/integration/test_lists/test-db/l0_gb300_multi_gpus_perf_sanity.yml
• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The GB300 DeepSeek-V4-Pro FP4 performance sanity test timeout increases from 90 to 120 seconds. The related skip waiver is removed.

Changes

GB300 performance sanity test

Layer / File(s)	Summary
Adjust test execution and waiver settings tests/integration/test_lists/test-db/l0_gb300_multi_gpus_perf_sanity.yml, tests/integration/test_lists/waives.txt	The test timeout increases to 120 seconds. The GB300 DeepSeek V4 Pro skip waiver is removed.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 30139

This enables the GB300 performance sanity test with a longer execution budget and removes its skip waiver. The configuration change is narrowly scoped and has no identified merge-blocking risk.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the NVBugs fix and the primary change: increasing the GB300 DeepSeek-V4-Pro con4301 performance-sanity budget to 120 minutes.
Description check	✅ Passed	The description explains the issue, rationale, scope, validation results, and checklist status. It does not use a separate Test Coverage heading, but it provides detailed hardware rerun evidence and t…
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[chenfeiz0326]

/bot run --stage-list "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-2,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-3,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-4,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-5"

[tensorrt-cicd]

PR_Github #72049 [ run ] triggered by Bot. Commit: 3013909 Link to invocation

[tensorrt-cicd]

PR_Github #72049 [ run ] completed with state FAILURE. Commit: 3013909
/LLM/main/L0_MergeRequest_PR pipeline #59107 (Partly Tested) completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[chenfeiz0326]

/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline"

[tensorrt-cicd]

PR_Github #72133 [ skip ] triggered by Bot. Commit: 3013909 Link to invocation

[tensorrt-cicd]

PR_Github #72133 [ skip ] completed with state SUCCESS. Commit: 3013909
Skipping testing for commit 3013909

Link to invocation