[AMD] ci: resolve the nightly image tag with git ls-remote instead of fetching tags

[michaelzhang-ai]

Motivation

Both AMD container-start scripts ran git fetch --tags origin for one reason: to learn the name of the latest release tag so they could spell the nightly image, rocm/sgl-dev:v0.5.18-rocm720-mi30x-<date>. They need the tag name and nothing the tag points at, but the fetch pulls every tag object and the history behind it into a checkout that actions/checkout created at depth 1. That is the single most expensive thing these scripts do before the image pull, and every AMD job pays it — a nightly runs ~90 of them concurrently against the same remote.

This is the tag-lookup half of #34487, split out on its own as requested. The image-tarball cache measured in that PR is not included here; the benchmark came out negative for it.

Modifications

• add scripts/ci/amd/amd_ci_latest_release_tag.py, which lists tag names with git ls-remote --tags --refs origin 'v*.*.*' and picks the highest. Tag ordering is imported from scripts/release/get_version_tag.py rather than reimplemented, so this resolves to the same tag the nightly release workflow published the image under (stable and post above rc). If the remote is unreachable it falls back to local tags; if it cannot order tags at all it returns nothing so the caller keeps its hard-coded default.
• point amd_ci_start_container.sh and amd_ci_start_container_disagg.sh at that helper. Both scripts carried a byte-identical copy of the old block and have been changed in lockstep by every prior fix to it ([AMD] Fix nightly version tag selection #23644, [Chore] Move version tag helper to release scripts #35196), so leaving one behind on the fetch would have half-solved the problem and invited drift.
• log how long the lookup took next to the resolved version, so the nightly logs stay measurable without a separate probe.
• add test/registered/unit/tools/test_amd_ci_latest_release_tag.py, covering the helper plus a guard that neither container script goes back to fetching tags.

No workflow YAML changes, and the flags both scripts accept are unchanged, so a workflow file from any ref still drives a checkout from any other ref. The helper and its caller always travel together in the same checkout.

scripts/release/get_version_tag.py is untouched: it is what setuptools-scm invokes via git_describe_command in six pyprojects, and nothing here needs it to change.

Accuracy Tests

N/A — CI setup plumbing, no model or kernel code.

Speed Tests and Profiling

Measured on a fresh git clone --depth 1, which is what actions/checkout@v4 leaves the AMD jobs (1 commit, no tags, shallow):

Step	Wall time	.git size after
depth-1 clone	47s	33MB
git ls-remote --tags --refs origin 'v*.*.*'	1s	33MB (unchanged)
git fetch --tags origin	29s	201MB

So the fetch transferred ~168MB of objects to answer a question ls-remote answers in one round trip and 150 lines of output. The 29s above is a well-provisioned VM with a fast link to GitHub; a second run against release/v0.5.18 took 125s, in line with the 136s median measured on the AMD runners in #34487, where the worst ls-remote lookup across all those runs was 20s.

The end-to-end helper on a depth-1 checkout with no local tags returns in 0.29s:

$ python3 scripts/ci/amd/amd_ci_latest_release_tag.py
v0.5.18

The dropped fetch changes no version that anything downstream reads. get_version_tag.py in describe mode — the mode setuptools-scm actually calls — fails identically before and after the fetch, because a single-commit HEAD has no common ancestor with any release tag:

# depth-1 checkout, no tags fetched
$ python3 scripts/release/get_version_tag.py
WARNING: No version tags (v*.*.*) found in repo
ERROR: Could not determine version from git tags. ... exit 1

# same checkout, after `git fetch --tags origin` brought in all 150 tags
$ python3 scripts/release/get_version_tag.py
WARNING: No common ancestor between v0.5.18 and HEAD. Is this a shallow clone?
ERROR: Could not determine version from git tags. ... exit 1

Either way setuptools-scm falls back to the same tagless dev version. The only thing the fetch enabled was --tag-only printing v0.5.18, which is exactly what the new helper returns.

Branch coverage

These scripts do not only run on main. amd-aiter-scout.yml drives all four AMD workflows at ref: amd/aiter-ci, the PR workflows check out inputs.pr_head_sha, and release branches get cherry-picks. Because git ls-remote asks the remote for its refs, the answer does not depend on which ref is checked out, and none of the AMD workflows override repository: or token: — so origin is always sgl-project/sglang. Old and new paths were compared on a depth-1 clone of each branch:

Branch checked out	ls-remote helper	fetch --tags + old helper	Agree
main	v0.5.18 (1s)	v0.5.18 (35s)	yes
amd/aiter-ci (aiter scout)	v0.5.18 (0s)	v0.5.18 (14s)	yes
release/v0.5.18	v0.5.18 (0s)	v0.5.18 (125s)	yes
release/v0.5.17	degrades to default	n/a — pre-#35196 layout	—

Note this resolves the newest release tag on every branch, including release branches, exactly as the fetch did; picking a branch-scoped version was never the behavior and is not changed here.

That last row is a gap the branch sweep found and this PR now handles. release/v0.5.17 predates #35196, so its ordering helper is still at python/tools/get_version_tag.py; loading it from the new path raised FileNotFoundError, and the caller's || true turned that into a bare traceback. The helper now reports the missing file and returns no tag, so the caller keeps its default the way it did before:

$ python3 scripts/ci/amd/amd_ci_latest_release_tag.py   # on release/v0.5.17
WARNING: .../scripts/release/get_version_tag.py is missing, so release tags
cannot be ordered the way the nightly image was published
ERROR: could not resolve a release tag

$ # through the caller's block, under `set -euo pipefail`
Warning: No version tags resolved; using default v0.5.5      (exit 0)

It deliberately does not guess an order in that case: without the shared helper, v0.5.10rc0 sorts above v0.5.10 under strverscmp, which is the bug #35196's helper exists to avoid, and that would name an image the nightly never published.

nightly-test-amd-miles-rocm720.yml passes --custom-image, so it discards the resolved tag entirely — it was paying the full fetch for a value it never used, and now pays ~1s. Skipping the lookup outright for --custom-image would mean reordering the argument parsing that builds the default base tags, which is not worth the churn for a second.

Verification

Fallback and failure paths, exercised on a depth-1 shallow checkout:

Condition	Result
remote reachable, no local tags	v0.5.18 in 0.29s
remote unreachable, local tags present	warns, falls back to local tags, shared ordering
remote unreachable, no tags at all	warns, empty stdout, exit 1
ordering helper missing from the checkout	warns with the path, empty stdout, exit 1
the caller's bash block on either of those	Warning: No version tags resolved; using default v0.5.5, exit 0 under set -euo pipefail

Also run:

• 7/7 unit tests in the new file, including a negative check that the anti-regression guard really fails when git fetch --tags is reintroduced into either script;
• scripts/lint/check_registered_tests.py and check_no_bare_pytest_main.py;
• bash -n on both container scripts;
• pre-commit on all four files.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations. — n/a, no user-facing surface
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #34829113888
Latest PR Test (Extra): ❌ Run #34829113577
Latest PR Test (AMD ROCm 10): ❌ Run #34829113943