[AMD][DO NOT MERGE] Fix MI35x DeepSeek-V3.2 nightly timeout failures

[michaelzhang-ai]

Motivation

The nightly MI35x DeepSeek-V3.2 tests (nightly-perf-8-gpu-mi35x-deepseek-v32-basic, nightly-accuracy-8-gpu-mi35x-deepseek-v32, etc.) started failing with exit code 255 (timeout) since Feb 21. Feb 20 and earlier were green.

Root Cause

MI35x Kubernetes pod was recycled, causing:

1. No persistent volume mount → CACHE_VOLUME="" in amd_ci_start_container.sh → all caches (HF models, MIOpen kernels, Triton kernels) go to ephemeral container storage
2. Every CI run must re-download models and recompile all JIT kernels from scratch, adding ~20-30 min overhead to every MI35x job
3. DeepSeek-V3.2 tests exceed their timeout because 400GB+ model download + kernel compilation + benchmarks > 90 min test timeout limit
4. Self-perpetuating failure: container is destroyed on timeout, partial downloads are lost, next run starts over

Evidence:

• ALL MI35x jobs got 2-3x slower starting Feb 21 (grok1: 21m→1h10m, kimi: 19m→48m, mxfp4: 18m→45m) - kernel cache recompilation overhead
• Only DSv32 jobs cross the timeout threshold due to the ~400GB model download on top of kernel recompilation
• MI30x jobs are unaffected (different runner, caches intact)
• PR [AMD] support two batch overlapping for mori ep #17953 code regression was reverted (PR Revert "[AMD] support two batch overlapping for mori ep #17953" #19161, Feb 22) but MI35x failures persist → confirms infrastructure root cause

Additionally, DSv32 accuracy tests hardcoded HF_HOME to /data2/models/huggingface (ephemeral pod-local storage), overriding the CI container's persistent volume path even when available.

Modifications

1. Remove hardcoded /data2 paths from 3 DSv32 MI35x accuracy test files so they inherit HF_HOME=/sgl-data/hf-cache from the CI container
2. Add model pre-cache step (snapshot_download with 240-min timeout) before each of the 8 MI35x DSv32 test jobs across both ROCm 7.0 and 7.2 workflows
   • Pre-cache checks if model is already cached (>400GB) before downloading
   • Logs cache status and download timing for debugging

Files changed

File	Change
nightly-test-amd.yml	Add pre-cache step to 4 MI35x DSv32 jobs + temp PR trigger
nightly-test-amd-rocm720.yml	Add pre-cache step to 4 MI35x DSv32 jobs + temp PR trigger
test_deepseek_v32_eval_mi35x.py	Remove /data2 HF_HOME override
test_deepseek_v32_mtp_eval_mi35x.py	Remove /data2 HF_HOME override
test_deepseek_v32_dp_eval_mi35x.py	Remove /data2 HF_HOME override

Note: pull_request triggers are temporary for CI testing and will be removed before merge.

Please help review. @yctseng0211, @bingxche.

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!