Skip to content

[AMD][CI] Temporarily bypass local-registry image pulls #36171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yctseng0211 merged 1 commit into from
Aug 24, 2026
Merged

[AMD][CI] Temporarily bypass local-registry image pulls #36171

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 4 additions & 22 deletions scripts/ci/amd/amd_ci_start_container.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,15 +158,7 @@ find_latest_image() {
fi
done

# If not found locally, fall back to pulling from public registry.
# We intentionally do not probe ${LOCAL_DOCKER_REGISTRY} here with
# `docker manifest inspect --insecure` because that command runs in the
# runner pod's network namespace, which on every observed AMD scale set
# cannot reach 10.44.14.109:5000 (every probe either fast-fails with TLS
# reject or hits a 30s TCP timeout, multiplied across 7 daily candidates).
# The actual local-registry pull still happens in the call site below via
# `docker pull "${LOCAL_DOCKER_REGISTRY}/${IMAGE}"`, which goes through the
# docker daemon on the host and inherits its insecure-registries config.
# If not found locally, resolve the latest tag from the public registry.
for days_back in {0..6}; do
image_tag="${base_tag}-$(date -d "${days_back} days ago" +%Y%m%d)"
echo "Checking for image: rocm/sgl-dev:${image_tag}" >&2
Expand Down Expand Up @@ -272,19 +264,9 @@ elif [[ -n "${BUILD_FROM_DOCKERFILE}" ]]; then
else
# Find the latest pre-built image
IMAGE=$(find_latest_image "${GPU_ARCH}")
# Try the local docker registry first (avoids Docker Hub rate limits and is
# faster on the LAN); if that fails for any reason, fall back to the
# public registry with exponential-backoff retries. Capture stderr so the
# real failure reason (TLS handshake, 404, connection refused, etc.) is
# visible in the job log instead of being silently swallowed.
if local_pull_output=$(docker pull "${LOCAL_DOCKER_REGISTRY}/${IMAGE}" 2>&1); then
echo "Pulled from local docker registry: ${LOCAL_DOCKER_REGISTRY}/${IMAGE}"
docker tag "${LOCAL_DOCKER_REGISTRY}/${IMAGE}" "${IMAGE}"
else
echo "Local docker registry pull failed; falling back to public registry: ${IMAGE}" >&2
printf '%s\n' "${local_pull_output}" | sed 's/^/ [local-pull] /' >&2
retry_with_backoff 6 docker pull "${IMAGE}"
fi
# Temporarily bypass the shared local registry while concurrent CI pulls
# saturate it. Keep using the authenticated, retried public-registry path.
retry_with_backoff 6 docker pull "${IMAGE}"
fi

CACHE_HOST=/home/runner/sglang-data
Expand Down
22 changes: 4 additions & 18 deletions scripts/ci/amd/amd_ci_start_container_disagg.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,11 +143,7 @@ find_latest_image() {
fi
done

# If not found locally, fall back to pulling from public registry.
# See amd_ci_start_container.sh for why we don't probe
# ${LOCAL_DOCKER_REGISTRY} with `docker manifest inspect --insecure` from
# the runner pod's network namespace; the actual local-registry pull
# happens at the call site below via the docker daemon on the host.
# If not found locally, resolve the latest tag from the public registry.
for days_back in {0..6}; do
image_tag="${base_tag}-$(date -d "${days_back} days ago" +%Y%m%d)"
echo "Checking for image: rocm/sgl-dev:${image_tag}" >&2
Expand Down Expand Up @@ -225,19 +221,9 @@ if [[ -n "${CUSTOM_IMAGE}" ]]; then
fi
else
IMAGE=$(find_latest_image "${GPU_ARCH}")
# Try the local docker registry first (avoids Docker Hub rate limits and is
# faster on the LAN); if that fails for any reason, fall back to the
# public registry with exponential-backoff retries. Capture stderr so the
# real failure reason (TLS handshake, 404, connection refused, etc.) is
# visible in the job log instead of being silently swallowed.
if local_pull_output=$(docker pull "${LOCAL_DOCKER_REGISTRY}/${IMAGE}" 2>&1); then
echo "Pulled from local docker registry: ${LOCAL_DOCKER_REGISTRY}/${IMAGE}"
docker tag "${LOCAL_DOCKER_REGISTRY}/${IMAGE}" "${IMAGE}"
else
echo "Local docker registry pull failed; falling back to public registry: ${IMAGE}" >&2
printf '%s\n' "${local_pull_output}" | sed 's/^/ [local-pull] /' >&2
retry_with_backoff 6 docker pull "${IMAGE}"
fi
# Temporarily bypass the shared local registry while concurrent CI pulls
# saturate it. Keep using the authenticated, retried public-registry path.
retry_with_backoff 6 docker pull "${IMAGE}"
fi

CACHE_HOST=/home/runner/sglang-data
Expand Down
Loading