Skip to content

[https://nvbugs/6676844][test] Unwaive test_wan_t2v_example after CI checkpoint storage fix - #18732

Merged
karljang merged 1 commit into
NVIDIA:mainfrom
chang-l:unwaive-6676844
Sep 8, 2026
Merged

[https://nvbugs/6676844][test] Unwaive test_wan_t2v_example after CI checkpoint storage fix#18732
karljang merged 1 commit into
NVIDIA:mainfrom
chang-l:unwaive-6676844

Conversation

@chang-l

@chang-l chang-l commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Dev Engineer Review

  • Removed the waiver for examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example from tests/integration/test_lists/waives.txt.
  • The change does not modify production code, public APIs, or test behavior.
  • The scope is limited to restoring normal execution of the Wan integration test.

QA Engineer Review

  • Modified tests/integration/test_lists/waives.txt.
  • Removed the waiver for examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example.
  • The test is listed in the CI files test-db/l0_dgx_b200.yml and test-db/l0_b200.yml.
  • The test passed on B200 when the waiver was removed. Later collection failures were unrelated main-branch import errors.
  • Coverage verdict: sufficient.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Removes the waiver for the Wan T2V integration test. The test remains listed in the appropriate B200 CI test lists.

Description

Unwaive examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example, waived in #18303 for https://nvbugs/6676844.

The failure was not a code regression. The example died during weight loading with Error while deserializing header: header too large on Wan2.2-T2V-A14B-Diffusers-NVFP4/transformer/diffusion_pytorch_model-00002-of-00002.safetensors on the CI model share (/scratch.trt_llm_data/llm-models). The share's disk filled up on 2026-08-26 and the shard was left corrupt: every other test in the same DGX_B200-PyTorch-4 stage passed (28 passed / 1 skipped / this 1 failure), the Qwen-Image example loaded fine through the same loader, and no visual_gen source changed between the last good commit (ca939b7) and the first bad one (128950d). The infra owner cleaned and repaired the share on 2026-08-27.

This PR only removes the waiver line. The test is in the pre-merge DGX_B200-PyTorch stage (l0_b200.yml), so this PR's CI verifies the NVFP4 checkpoint loads again.

Test Coverage

  • examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example (l0_b200.yml pre_merge; also listed in l0_dgx_b200.yml post_merge)

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@chang-l

chang-l commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 87c2931f-1d8b-4210-a98f-e40d31221a57

📥 Commits

Reviewing files that changed from the base of the PR and between 0369d45 and b5635e2.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


Walkthrough

The integration waiver list no longer skips test_wan_t2v_example in examples/visual_gen/test_visual_gen_wan.py.

Changes

Wan test execution

Layer / File(s) Summary
Remove Wan waiver
tests/integration/test_lists/waives.txt
Removed the waiver and associated bug reference for the Wan text-to-video example test.

Priority: ⬇️ Low — Defer this one-line test-waiver removal because it is a narrow validation change for the Wan text-to-video example after checkpoint storage repair.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to b5635

The change re-enables the Wan text-to-video example test in the integration waiver list. No current merge-blocking risk remains.

Suggested reviewers: sylvesterkaczmarek

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the NVBugs ticket, test-related change, specific test, and reason for removing the waiver.
Description check ✅ Passed The description explains the failure cause, infrastructure resolution, scope of the change, and relevant pre-merge test coverage. It includes the required sections and checklist.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #71566 [ run ] triggered by Bot. Commit: e47637f Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #71566 [ run ] completed with state SUCCESS. Commit: e47637f
/LLM/main/L0_MergeRequest_PR pipeline #58652 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chang-l

chang-l commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

CI status note for L0 #58652 (this PR's first run):

  • The unwaived test passed on B200: CBTS narrowed the DGX_B200-PyTorch stage to exactly examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example and it reported 1 passed (call 511 s, exit code 0). So the Wan2.2 NVFP4 checkpoint on the CI share is restored and the waiver can go.
  • The pipeline FAILURE comes from CPU-Generic-x86-1, unrelated to this change: unittest/others (test_time_breakdown.TestRequestTimeBreakdown::test_parse_records_matches_parse_json_file, Lists differ) and unittest/llmapi/apps/test_request_metrics.py. Both tests were added by [None][feat] perf-sanity: upload per-request disagg lifecycle spans to OpenSearch #18445 (737af47, merged a few hours before this branch), and the same two failures show up today in every other PR run whose base includes that commit, so this is a main-branch breakage. I'll rebase and re-run once a fix or waiver for those lands on main.

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the waiver is the right recovery test because the original failure was attributed to corrupted shared checkpoint storage rather than a source regression. Since this PR changes only test admission, I would treat the exact pre-merge B200 execution of test_wan_t2v_example as the merge gate; that proves the repaired checkpoint path actually loads under the same environment that failed before. No code-path risk beyond that validation.

@coderabbitai

coderabbitai Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@chang-l

chang-l commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #71779 [ run ] triggered by Bot. Commit: ffe25c2 Link to invocation

@chang-l

chang-l commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main (70feda6395, includes the #18445 revert) and re-ran: L0 #58858 now fails before any test stage, in Check Test List (15 pytest collection errors under tests/integration/defs). This is the main-wide import breakage tracked by #18771 (kv_cache_manager_v2 moved by #17899 while #17399 still imports the old path, so tensorrt_llm._torch.attention.backends cannot import). Every single-GPU sub-job started after ~14:00 UTC today fails the same way. Will rebase and re-run once that fix lands; the target test itself already passed in the first run (L0 #58652).

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #71779 [ run ] completed with state SUCCESS. Commit: ffe25c2
/LLM/main/L0_MergeRequest_PR pipeline #58858 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@sylvesterkaczmarek

Copy link
Copy Markdown

Thanks for tracing this. Since test_wan_t2v_example passed in the earlier run and the current rerun is failing during collection because of the main-wide import break tracked in #18771, I agree this does not point to a regression in this waiver removal. Happy to recheck once the upstream collection issue is fixed and the target test can run again on the rebased head.

…checkpoint storage fix

The failure was not a code regression: the Wan2.2-T2V-A14B-Diffusers-NVFP4
checkpoint on the CI model share had a corrupt safetensors shard
(transformer/diffusion_pytorch_model-00002-of-00002.safetensors, 'header too
large') after the llm-models scratch disk filled up on 2026-08-26. The infra
owner cleaned and repaired the share on 2026-08-27, so the waiver added in
NVIDIA#18303 can be dropped.

The test runs in the pre-merge DGX_B200-PyTorch stage (l0_b200.yml), so this
PR's CI run verifies that the NVFP4 checkpoint loads again.

Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
@chang-l

chang-l commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@coderabbitai

coderabbitai Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #72190 [ run ] triggered by Bot. Commit: b5635e2 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #72190 [ run ] completed with state SUCCESS. Commit: b5635e2
/LLM/main/L0_MergeRequest_PR pipeline #59232 completed with status: 'SUCCESS'

CI Report

Link to invocation

@chang-l

chang-l commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

@sylvesterkaczmarek the rebased head (b5635e20f5, on top of main with #18771) is now green end to end: L0 #59232 completed with status SUCCESS. Check Test List passed, and the CBTS-narrowed DGX_B200-PyTorch-1-cbts stage again ran exactly examples/visual_gen/test_visual_gen_wan.py::test_wan_t2v_example1 passed (call 516 s, exit code 0), matching the first run. So the repaired checkpoint has now loaded twice in the same pre-merge B200 environment that failed before. Ready for review.

@karljang karljang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM;

@karljang
karljang merged commit c495ad1 into NVIDIA:main Sep 8, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants