[XPU CI] Optimize PR CI per-file timeouts and fail hanging tests fast

[arathi-hlab]

Motivation

XPU PR CI ran each test file with the default 1200s limit plus 600s on retry. Several files intermittently hang right after XPU device init with no output, so each hang burned the full 30 min before a retry that passes in seconds. Across 156 recent XPU stage-a jobs there were 45 such hangs.

Split out of #41857 so that PR only carries the nightly kernel-main/release split.

Modifications

• pr-test-xpu.yml: cap every file at 600s for both attempts (--timeout-per-file 600 --retry-timeout-increase 0), and pass per-file overrides for the 7 files seen hanging.
• run_suite.py: add a repeatable --timeout-override PATH=SECONDS, matched by path suffix. It does nothing unless passed, and only pr-test-xpu.yml passes it.
• ci_utils.py: an override caps only the first attempt. The retry gets the full --timeout-per-file, so a slow-but-healthy run still gets a fair second chance.

Overrides are ~3x each file's observed max passing runtime:

File	Normal	Hangs seen	First-attempt timeout
xpu/test_xpu_basic.py	21-41s	7	180s
kv_canary/test_self_unit_req_to_expected_token_ids_manager.py	8-74s	3	120s
kernels/ops/kvcache/test_hisparse.py	6-9s	11	120s
kv_canary/test_self_unit_plan_input.py	7-11s	6	120s
xpu/test_layernorm_xpu.py	7-11s	4	120s
e2e/xpu/test_xpu_fp8_linear.py	8-72s	7	240s
xpu/test_intel_xpu_backend.py	58-112s	7	360s

A hang on one of these now costs about 3 min (cap + 60s wait + normal retry) instead of about 31 min.

No test file or register_*_ci change, so other vendors' CI behaviour is unchanged.

Testing

• Ran run_unittest_files on dummy tests with a 2s override and a 10s base: attempt 1 was killed at 2s, the retry ran 3s (longer than the override) and passed; a non-hanging overridden file passed on attempt 1.
• --timeout-override rejects x.py, x.py=0, =5, x.py=abc.
• pre-commit (isort, ruff, ruff format, codespell, check-yaml) passes.

---

CI States

Latest PR Test (Base): ❌ Run #38026054009
Latest PR Test (Extra): ❌ Run #38026053877
Latest PR Test (AMD ROCm 10): ❌ Run #38026054002

[mingfeima]

The separation looks right: XPU-specific timeout values stay in the XPU workflow, while the shared runner behavior is opt-in. The XPU PR job passes at this head.

• python/sglang/test/ci/ci_utils.py:269-273,345-358: Could you add a hardware-independent regression test for a matched first attempt, its retry using the base timeout, and an unmatched file? The passing XPU job did not hit a timeout, so it does not exercise the new retry behavior. Since this is described as a cap, min(base_timeout, override) would also prevent an override from accidentally extending the first attempt.
• test/run_suite.py:440-448: Please reject non-finite values with math.isfinite(timeout). Both nan and inf currently pass the positivity check and can interfere with timeout handling.

[arathi-hlab]

@mingfeima Thanks! Addressed in b1de55a:

• Applied min(base_timeout, override) so an override can only shorten the first attempt.
• Added a math.isfinite() check so --timeout-override rejects nan/inf.

For the retry path I verified locally with dummy tests (10s base, 2s override): attempt 1 was killed at 2s, the retry got the full 10s, ran 3s (longer than the override) and passed; a non-hanging overridden file passed on attempt 1, and an unmatched file used the base timeout. I kept it out of the suite since it's CI-runner tooling with no existing test harness.

[arathi-hlab]

/tag-and-rerun-ci