[mickqian]

Motivation

perf_repeat_requests issued multiple requests but validated only the last output. This can hide a failing first request or leave request-to-request state corruption insufficiently checked. #38171 is an example of why repeated requests need coverage.

Modifications

• Validate every repetition, including performance, load/runtime peak memory, video/image consistency, and audio consistency. Keep checking subsequent requests and aggregate failures without allowing a later pass or skip to hide a failure.
• Keep H3 T2VA and ref2va as separate checkpoint-partition cases. Each sends two 4-second, 8-step requests to the same server. T2VA exercises VAE layerwise offload; ref2va retains resident VAEs.
• Enable ref2va performance checks using the same-workload H100 measurements from run 34006833375. Keep sampling parameters, GT, consistency thresholds, and existing T2VA performance thresholds unchanged.
• Preserve request indices in reports and isolate failure artifacts. Baseline generation retains the worst observed metric across repetitions.
• Replace class-level result accumulation and dynamically injected pytest-config attributes with per-case and session fixtures. Remove the cross-import of conftest and debug prints; keep failed-case results through teardown.
• Document sequential checks and add regression tests. Video consistency failure no longer prevents the independent audio GT check.

Rebased onto 67e3ccda97e0e, which includes #38171. No production runtime changes: all changes are tests and documentation.

Accuracy Tests

Final head 9f1a1af9ef622bd29d87e4e1fa836914d9c2b645, isolated H200 environment:

test_sequential_server_requests.py
test_performance_metrics.py
test_suite_partitioning.py
62 passed
pre-commit: passed

The fixture integration test uses a real pytest subprocess, two test classes, and both passing and failing cases. It checks that all eight request records survive without leaking case-local state.

Real-server validation is not fully green. Prior to the fixture cleanup/rebase, both H3 cases completed both requests on 2x H200 with performance and VRAM checks enabled. Against the existing H100 video GT:

Case	Request 1 min SSIM	Request 2 min SSIM	Threshold
H3 T2VA	0.8971	0.8971	0.92
H3 ref2va	0.8200	0.7280	0.92

A resident-VAE control reproduced these video metrics; independent audio GT checks passed for all four control requests. Ref2va also fails CLIP on the second request. The cross-request output difference still needs investigation. H200 results do not establish whether H100 CI passes, and no GT or consistency thresholds were changed to hide these failures.

Speed Tests and Profiling

Pre-rebase 2x H200 measurements at 3dad316967e21, Python 3.12, PyTorch 2.13.0+cu130, Transformers 5.12.1, 4 seconds / 8 steps:

Case	Request 1 E2E	Request 2 E2E	Load peak	Runtime peak, requests 1 / 2
H3 T2VA, VAE layerwise offload	19.899 s	19.824 s	3336 MiB	24838 / 24840 MiB
H3 ref2va, resident VAE	65.893 s	65.454 s	15242 MiB	34560 / 34560 MiB

All four requests passed the configured performance/VRAM checks. These are coverage measurements, not a claimed inference speedup or H100 baseline calibration. Existing server startup is reused; the two H3 cases each add one short request. Partition time estimates were increased accordingly.

Checklist

• Run pre-commit.
• Add unit and pytest fixture lifecycle tests.
• Update documentation.
• Report real-server results and known accuracy failures.
• Keep production runtime and GT unchanged.

---

CI States

Latest PR Test (Base): ✅ Run #34083939361
Latest PR Test (Extra): ❌ Run #34083939239
Latest PR Test (AMD ROCm 7.2): ⏳ Run #34083939346