[tzhouam]

Contributor intent (required human answers)

1. What outcome do you want to achieve?

Stop the READY blocking job CosyVoice3-TTS E2E Smoke (SDPA) on AMD CI from failing with a stage-init timeout whenever the modelscope mirror is slow, so release backports such as #9470 are not blocked by an infra flake. (Explicit request from @tzhouam while triaging the AMD failure on #9470.)

2. What is the highest-leverage variable for that outcome?

Pending contributor answer.

3. How will this work move that variable?

Pending contributor answer.

4. Whose customer or user needs does this address, and what are those needs?

Pending contributor answer.

Purpose

Before. CosyVoice3 calls s3tokenizer.load_model("speech_tokenizer_v3_25hz") while the LLM stage initializes. On a cold cache that is a ~925 MiB download from modelscope.cn into $XDG_CACHE_HOME/s3tokenizer. .buildkite/amd/scripts/run-amd-test.sh points XDG_CACHE_HOME at /tmp/vllm-omni-native-<job_id>/xdg, which is discarded with the pod, so every CosyVoice3 AMD job re-downloads the checkpoint inside the --stage-init-timeout 600 window. When the mirror serves at ~2 MiB/s the download alone takes 8–10 minutes and the job fails as an opaque timeout:

Downloading onnx checkpoint:  55%|█████▍ | 505M/925M [09:39<03:48, 1.93MiB/s]
[StageInit] LLM stage-0 replica-0 not READY after 600s (stage_init_timeout); terminating its processes
ERROR tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py::test_voice_clone_zh_002[cosyvoice3_async_chunk]

Seen today on unrelated PRs at the same time: AMD builds 14817 (#9470), 14814 (#9463), 14815 (#9174), 14801 (#9377), 14802 (#9335). In 14814 the download finished at 8m49s and the job still timed out during JIT warmup.

After.

• run-amd-test.sh symlinks $XDG_CACHE_HOME/s3tokenizer → $HF_HOME/s3tokenizer, so the checkpoint lives on the persistent Hugging Face volume (the same one VLLM_CI_REQUIRE_PERSISTENT_HF_CACHE=1 already guarantees) and is fetched once per node instead of once per job.
• New .buildkite/amd/scripts/prefetch_s3tokenizer.py downloads the checkpoint (3 attempts; s3tokenizer re-downloads on SHA-256 mismatch, so a retry recovers an interrupted transfer) before pytest in the three CosyVoice3 AMD jobs (CosyVoice3-TTS E2E Smoke (SDPA) in READY, CosyVoice3-TTS E2E Test and CosyVoice3 Offline Zero-shot Smoke in nightly). A slow or dead mirror now costs job wall time and surfaces as an explicit download failure, never as an engine stage-init timeout.

No runtime code changes; vllm_omni/ is untouched. The touched files are identical on release/v0.32.0, so this cherry-picks cleanly there as a follow-up.

Test Plan

vLLM Version: n/a (CI configuration only)

vLLM-Omni Commit: a3aba86

ruff check .buildkite/amd/scripts/prefetch_s3tokenizer.py tests/buildkite/
ruff format --check .buildkite/amd/scripts/prefetch_s3tokenizer.py tests/buildkite/test_prefetch_s3tokenizer.py tests/buildkite/test_amd_pipeline.py
python3 tools/pre_commit/check_test_marks.py tests/buildkite/test_prefetch_s3tokenizer.py tests/buildkite/test_amd_pipeline.py
bash -n .buildkite/amd/scripts/run-amd-test.sh
python3 -m pytest -q tests/buildkite/test_amd_pipeline.py tests/buildkite/test_prefetch_s3tokenizer.py tests/buildkite/test_amd_nightly_source_filter.py tests/buildkite/test_amd_ready_r2_01.py

Plus the AMD CI run on this PR: CosyVoice3-TTS E2E Smoke (SDPA) should now log s3tokenizer checkpoint speech_tokenizer_v3_25hz ready in …s before pytest starts.

Test Result

• 94 passed for the four tests/buildkite modules above (new: test_amd_runner_keeps_s3tokenizer_cache_on_persistent_volume, prefetch ordering asserted in the three CosyVoice3 step tests, 5 behavior tests in test_prefetch_s3tokenizer.py covering success, retry-then-succeed, exhausted retries, invalid attempts and CLI parsing).
• ruff check/ruff format --check, check_test_marks.py and bash -n clean.
• Symlink wiring smoke-tested locally with a throwaway HF_HOME/XDG_CACHE_HOME pair: ln -sfn is idempotent and s3tokenizer's default download_root resolves through the link.
• Not run locally: the AMD jobs themselves (need an MI300 pod); relying on Buildkite for that.