[https://nvbugs/6544407][fix] unwaive glm5 stress test

[chuangz0]

Dev Engineer Review

• Removed the waiver entry for the B200 GLM5 NVFP4 TP4/EP4/DP stress test.
• The change is limited to tests/integration/test_lists/waives.txt.
• The change has no code, API, performance, or error-handling impact.
• The waiver removal is consistent with the reported inability to reproduce the error on main.
• The test-list format and bug-reference consistency require verification against the file contents.

The error was not reproduced after running the test five times on the b200 x8 node，
link https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/job/DEBUG_LLM_FUNCTION_CLUSTER_TEST/1845/

QA Engineer Review

• Modified test-list file: tests/integration/test_lists/waives.txt.
• Removed the B200 GLM5 NVFP4 TP4/EP4/DP stress-test waiver.
• No test-db or QA entries were added or modified.
• Verdict: needs follow-up until CBTS coverage data confirms that the test runs without the waiver.

Description

unable to reproduce the error on latest main

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

Walkthrough

The integration waive list removes the skip entry for the B200 GLM5 NVFP4 TP4/EP4/DP stress test.

Changes

Cohort / File(s)	Summary
Waive list update tests/integration/test_lists/waives.txt	Removes the waived skip entry for the B200 GLM5 NVFP4 TP4/EP4/DP stress test.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: schetlur-nv, qijune

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description states the reproduction result but leaves Test Coverage empty and does not explain the change or validation in enough detail.	Add the relevant GLM5 stress-test validation and briefly explain why removing the waiver resolves the issue.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the NVBugs fix and the removal of the GLM5 stress-test waiver.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[chuangz0]

/bot run

[tensorrt-cicd]

PR_Github #63916 [ run ] triggered by Bot. Commit: fd85d6f Link to invocation

[brnguyen2]

The deletion is correct and complete — the removed entry matches the pytest node ID exactly, and no duplicate or leftover waiver for this test exists elsewhere in test_lists.

The gap is evidence. This test is listed only in tests/integration/test_lists/qa/llm_function_stress.txt:8, not in any test-db/l0_*.yml, so /bot run on this PR will not execute it — nothing in CI can confirm the unwaive. Combined with what the test does (tests/integration/defs/disaggregated/test_disaggregated.py:2807: 8×B200, 35k requests at concurrency 512, 0.90 accuracy gate, 10% request cancellation), "unable to reproduce the error on latest main" is thin: that profile fails intermittently, which is usually why it gets waived in the first place.

Before merging, please add to the description: how many runs of disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-glm5_nvfp4_tp4_ep4_dp_stress] you did on main, and a link to at least one passing run. Also state the disposition of https://nvbugs/6544407 — if the failure is genuinely gone the bug should be closed (or closed as not-repro) in the same pass, otherwise the waiver and the bug drift apart and the next person hitting it has no trail. If the root cause was actually fixed by a specific commit, naming it would make this a one-line approve.

[brnguyen2]

Worth noting for whoever reviews the QA side: the two sibling stress waivers here (deepseek_r1_v2_fp4_stress and qwen3_32b_fp8_stress, both https://nvbugs/6472256) stay in place, so this unwaive is genuinely scoped to the GLM5 case — good. But since none of these run in pre-merge CI, the first signal that the GLM5 unwaive was premature will be a red QA stress campaign. If you're not confident the underlying issue is fixed (vs. just not reproducing), consider holding this until a couple of clean QA stress runs land rather than using the campaign as the experiment.

[BowenFu]

Holding on this one — I don't think the evidence retires the risk yet.

nvbugs/6544407 is still open P0 (Dev - Open - To fix, Open issue), and the recorded failure is a hard segfault, not a flaky assertion: worker_ctx_0.log ends in nixlUcxSharedThread::run -> nixlUcxWorker::arm -> ucp_worker_arm -> uct_cuda_base_iface_event_fd_arm -> cuEventQuery, followed by MPI worker rank 0 exited unexpectedly. For a race in a progress thread, one non-reproduction is weak evidence — the bug comment is just "failed to reproduce the error" with no hardware, commit, or iteration count.

The triage on the bug also names a concrete, still-unapplied mitigation: the run starts NixlTransferAgent with num_threads 8 (the default at _torch/disaggregation/native/transfer.py), and you capped TRTLLM_NIXL_NUM_THREADS=1 across the disaggregated unit tests in #16708 for exactly this reason — this integration stress case never got that cap. AIPerf logged 3,439 RequestCancellationError before the crash, so cancellation racing arm() is the stated suspect.

Two ways I'd be happy to approve:

1. Cite the run — GPU/board, commit, and how many stress iterations passed clean; or
2. Land the TRTLLM_NIXL_NUM_THREADS=1 cap (or whatever the real fix is) alongside the unwaive, so the entry comes back with the root cause addressed rather than on non-reproduction.

Nothing here is blocking on my side beyond that evidence — the diff itself is a one-line removal of a full:B200/ post-merge entry.

[tensorrt-cicd]

PR_Github #63916 [ run ] completed with state SUCCESS. Commit: fd85d6f
/LLM/main/L0_MergeRequest_PR pipeline #51853 completed with status: 'SUCCESS'

CI Report

Link to invocation

[crazydemo]

Review summary - CONCERNS

Verdict: The one-line deletion is correct and matches the pytest node ID exactly, but it cannot merge as-is (mergeable_state: dirty — rebase needed) and, more importantly, the unwaive is unverifiable by this PR's CI, so it should not be trusted without QA evidence.

Issues

• [MAJOR] tests/integration/test_lists/waives.txt:165 - unwaive not verifiable by CI; intermittent stress test may regress

QA view

• Test coverage: missing - the unwaived test lives only in qa/llm_function_stress.txt, not in any test-db/l0_*.yml, so /bot run on this PR does not execute it. Nothing here can confirm the test passes.
• SM coverage: touches B200 (sm100/nvfp4); this PR runs nothing on that arch for this test. Verification depends entirely on out-of-band QA cluster runs.
• Test code: this is an unwaive, not a fix — the intermittent failure itself is not addressed in code.
• Test time: unknown - not added to any l0 list, so CI runtime is unchanged; QA-run duration cannot be inferred from the diff.
• Needs /qa-verify: yes - a human QA must re-run the glm5 stress case on 8xB200 multiple times and confirm reliable passing, plus confirm the disposition of the bug.

Does this actually fix nvbugs/6544407?

Partial / unclear. The PR removes the SKIP waiver on the basis of "unable to reproduce on latest main" after 5 manual runs, but names no root-cause fix commit and does not state whether the bug is closed. For an 8xB200 / 35k-request / conc512 / 0.90-accuracy / 10%-cancellation stress test, intermittent failure is the norm and is why such tests get waived. The failing path is therefore not demonstrably closed — it may simply not have recurred in 5 runs.

Possible new issues

• Re-exposes the intermittent failure with no CI gate: the next scheduled QA stress run can flake exactly as before, now un-waived, with no trail back to this change.
• Waiver and bug drift apart: waiver removed while nvbugs/6544407 disposition is unrecorded, leaving the next person hitting it without context.

What I could not verify

• The actual pass/fail history on main — only the description's claim and a Jenkins link are given; I cannot see run results.
• Whether a specific commit fixed the root cause (none is named).
• The current state of nvbugs/6544407.

Automated review by NVCortex Lite, run by @crazydemo.

[crazydemo]

[MAJOR] Unwaive not verifiable by CI; intermittent stress test may regress

The deleted line unwaived test_disaggregated_stress_test[...glm5_nvfp4_tp4_ep4_dp_stress], but this test appears only in the QA stress list (tests/integration/test_lists/qa/llm_function_stress.txt), not in any test-db/l0_*.yml. Concretely: run /bot run on this PR and the test never executes, so CI cannot confirm the unwaive — the only evidence is 5 manual runs on a b200 node for a workload (8xB200, 35k requests, conc512, 0.90 accuracy gate, 10% cancellation) whose known failure mode is intermittent. Trigger for regression: the next scheduled QA stress run flakes exactly as before, now un-waived, and fails the suite with no CI trail pointing back here. Fix: before merging, (1) link at least one passing run and state the run count, and (2) resolve nvbugs/6544407 explicitly (close / close-not-repro) so the waiver and bug do not drift apart. If a specific commit fixed the root cause, name it in the description.

[crazydemo]

Review summary - Approve (non-blocking)

Approving so this is not blocked on me. The points raised in my review comment above are non-blocking — please read them and address what you agree with before merging.

Worth doing before this is relied on: Unwaive of an intermittent stress test with no CI coverage on this PR; a human QA must re-run the glm5 stress case on 8xB200 and confirm it passes reliably (multiple runs) before this is trusted, and confirm the disposition of nvbugs/6544407.

Automated review by NVCortex Lite, run by @crazydemo.