[AMD][CI] Fix ROCm 7.0's dead apt index fail the MORI dependency install

[michaelzhang-ai]

Motivation

Fixes cluster R387 from the Aug-20 AMD CI monitor report: all 27 jobs in PR Test ROCm 7.0 (AMD) run 32399046576, plus the nightly-7.0 kimi-k26 job, died identically in "Install dependencies".

The report filed this as pure infra needing an AMD artifactory restore. It is also a code regression, it is specific to ROCm 7.0, and the code half is fixable here. From the failing job log:

Err:13 http://compute-artifactory.amd.com/artifactory/list/rocm-osdb-22.04-deb compute-rocm-rel-7.0/38 amd64 Packages
  404  Not Found [IP: 10.216.51.87 80]
Fetched 48.0 MB in 3s (18.2 MB/s)
E: Some index files failed to download. They have been ignored, or old ones used instead.
##[error]Process completed with exit code 100.

Where it came from

The apt-get update is new. It was added on 2026-08-20 by c747822 (#30984, the ROCm 7.2.4 / Python 3.12 / torch 2.11 upgrade), inside a docker exec ... bash -c that runs under set -euo pipefail:

git submodule update --init --recursive
apt-get update
apt-get install -y --no-install-recommends libgrpc++-dev 2>/dev/null || true

Every other change that PR made to this script is gated on the image flavor — the extras swap (dev_hip → dev_hip_rocm724), the AITER torch.Stream patch, and the AITER commit lookup all sit behind IMAGE_STAGE_SUFFIX == "-rocm724". These two apt lines are the only ungated change in the PR, so a 7.2.4 fix also began running on ROCm 7.0. Before that commit the MORI reinstall did no apt work at all and was immune.

Why only ROCm 7.0

Not because the code path is 7.0-specific — the identical path runs on 7.2.4 and passes. It is purely base-image lineage:

Flavor	Base image	rocm-osdb source
7.0	rocm/sgl-dev:rocm7-vllm-20250904 (AMD-internal dev image)	yes
7.2.0	rocm/pytorch:rocm7.2_ubuntu22.04_py3.10_pytorch_release_2.9.1	no
7.2.4	rocm/pytorch:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0	no

Only 7.0 builds on an AMD-internal image, which is what drags in compute-artifactory.amd.com/.../rocm-osdb-22.04-deb. The dead index name encodes it exactly: compute-rocm-rel-7.0/38 is the internal ROCm 7.0 release channel, build 38.

Confirmed against a real 7.2.4 job from the same day (96420374950), which runs the same MORI reinstall and the same apt-get update and succeeds, fetching 40.1 MB cleanly. Its apt hosts are entirely public, with no compute-artifactory.amd.com entry. (7.2.0 is inferred from the same public lineage rather than observed — nothing currently runs it, since the PR gate defaults to rocm724.)

Why the guard is not just about rocm-osdb

That step reaches six external apt hosts:

http://compute-artifactory.amd.com     ← the one that died
https://apt.llvm.org
https://archive.ubuntu.com
https://packages.broadcom.com
https://ppa.launchpadcontent.net
https://repo.radeon.com

apt-get update returns non-zero if any index fails, so under set -e each of those six is a single point of failure for the whole AMD fleet — for a package that is optional. This is not hypothetical: R391, the cluster that turned the AMD docker build red the same day, was an apt.llvm.org GPG/curl failure, also on that list.

Note too that apt itself treated the failure as recoverable — "They have been ignored, or old ones used instead" — and every index we actually use fetched fine. Nothing in this repo installs from rocm-osdb; it is inherited from the base image.

Modifications

scripts/ci/amd/amd_ci_install_dependency.sh only. Two changes, addressing the root cause and the blast radius separately:

1. Run the apt step only on rocm724. Only that base actually lacks the package — its job log shows libgrpc++-dev "previously unselected", downloaded from noble/universe and unpacked. ROCm 7.0 and 7.2.0 built MORI without it for months before #30984, so there the apt round trip is pure risk, and 7.0 is precisely the image carrying the dead source. This restores pre-#30984 behaviour on 7.0/7.2.0 rather than merely tolerating the failure there.

2. Keep both apt steps non-fatal where they do run, and log which one degraded:

if [ '${IMAGE_STAGE_SUFFIX}' = '-rocm724' ]; then
  apt-get update || echo '[MORI] apt-get update reported errors; continuing with the indexes it did fetch'
  apt-get install -y --no-install-recommends libgrpc++-dev || echo '[MORI] libgrpc++-dev unavailable; building MORI without it'
fi

The gate alone would fix the reported incident but leave 7.2.4 exposed to the other five hosts; the guard alone would tolerate a failure that 7.0 and 7.2.0 need not risk at all. Dropping 2>/dev/null in favour of an explicit message keeps a degraded install visible instead of swallowing it.

No retry logic is added: the image already sets Acquire::Retries "5" via /etc/apt/apt.conf.d/80-net-hardening (visible in the log as the repeated Ign:13 attempts), and a 404 is not retryable.

This was the only unguarded apt-get update left in the AMD CI path; every other one under scripts/ci/ is already || true or wrapped in a retry loop, matching scripts/ci/cuda/ci_install_dependency.sh.

test/registered/unit/tools/test_amd_ci_install_dependency.py — a 45-line guard asserting that no apt-get call in this script can abort the run. It runs on base-a-test-cpu (no docker, no ROCm, no GPU) alongside the existing CI-tooling tests in test/registered/unit/tools/.

It carries more weight than it looks, because no pull-request-triggered workflow can catch this regression. pr-test-amd.yml (ROCm 7.0) was demoted to a daily shadow in #34204 and now has only schedule / workflow_dispatch / workflow_call triggers — no pull_request. The PR gate is pr-test-amd-rocm720.yml, which defaults to rocm724. So the image carrying the dead index is never exercised pre-merge, and the static check is the only pre-merge signal on this invariant.

Not addressed here

Removing the rocm-osdb source from the ROCm 7.0 image itself would be the cleanest cleanup, but it needs a docker/rocm.Dockerfile change plus an image rebuild — and release-docker-amd-nightly is currently red (R391), so it could not ship now. It would also only remove one of the six hosts. Worth a separate PR once the image pipeline is healthy.

Accuracy Tests

Not applicable — CI dependency-install script and its test only; no runtime or model code touched.

Speed Tests and Profiling

Not applicable.

Verification

The unit test fails against the pre-fix script, naming the offending line, and passes against the fix:

# against the pre-fix script
AssertionError: Lists differ: ['apt-get update'] != []

# against this branch
1 passed

The flavor gate was verified by expanding it for each IMAGE_STAGE_SUFFIX value against a stub apt-get that exits 100, checking both whether apt ran and whether the MORI build after it is still reached:

PASS  flavor='<7.0>'      apt ran: no   build reached: yes
PASS  flavor='-rocm720'   apt ran: no   build reached: yes
PASS  flavor='-rocm724'   apt ran: yes  build reached: yes

While developing the original guard I also replayed the incident end to end — a stub apt-get reproducing the 404 and exit 100, driving the apt lines extracted from the script inside the same set -euo pipefail context — confirming the MORI build is still reached both when libgrpc++-dev remains resolvable and when it does not, with no regression on the healthy path.

Also run: the full test/registered/unit/tools/ directory (12 passed), run_suite.py collection with sanity_check=True plus validate_all_suites over all registered files (clean; the new test resolves into base-a-test-cpu), shellcheck -S warning (no new findings — the only remaining warning is a pre-existing SC2034), bash -n, black, isort, ruff, codespell, and the check-registered-tests / check-no-bare-pytest-main lint hooks.

Note for reviewers: this PR's own AMD CI cannot reproduce R387 — it runs 7.2.4, whose public base has no rocm-osdb source. In-situ validation needs the daily pr-test-amd.yml cron (17:30 UTC) or a manual workflow_dispatch of it.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations. — n/a.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed. — n/a.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.

---

CI States

Latest PR Test (Base): ❌ Run #32442764695
Latest PR Test (Extra): ❌ Run #32442764522
Latest PR Test (AMD ROCm 7.2): ❌ Run #32442764708