CI: automate release wheel publishing (poller + nightly job reuse)

[Alexey-Rivkin]

What?

• nixl-build-wheel-release-poller (new): every 4 hours, finds release-branch commits with no published wheels in Artifactory and triggers nixl-ci-build-wheel-nightly per missing CUDA variant, with PUBLISH_DIR=release/<ver>.
• The nightly job itself is untouched here - its side of this work merged separately in CI: enhance nightly wheel build (spcx/Infinia plugins, ci_refspec, PUBLISH_DIR) #2151 and ci: streamline nixl-ci-build-wheel-nightly parameters and tag wheels with build provenance #2166.

Published wheels: https://artifactory.nvidia.com/artifactory/sw-nbu-swx-nixl-pypi-local/release/

Why?

Release wheels are published manually today. With this, every merge to a release branch publishes automatically (HPCINFRA-4573).

How?

• Fully dynamic: scans all release/* branches with version >= 1.4.0, auto-detected, no per-release config.
• Triggers with NIXL_VERSION, CUDA_MAJOR, PUBLISH_DIR and ci_refspec only. UCX and the bundled plugins are not passed - each release ships what its own contrib/build-container.sh pins.
• Skips a release whose build-container.sh predates --build-options-file, since the nightly always passes it and older scripts reject unknown options. Release branches cut after ci: streamline nixl-ci-build-wheel-nightly parameters and tag wheels with build provenance #2166 pass on their own.
• No marker files: wheel presence per CUDA variant decides. A failed build retries next cycle; delete the folder in Artifactory to force a rebuild.
• Pre-merge e2e testing: run the poller with ci_refspec=refs/pull/<n>/head - it forwards the ref to the builds it triggers.

Summary by CodeRabbit

• New Features
  • Added a scheduled release poller that scans release branches for missing CUDA wheel artifacts and triggers builds only for the specific missing variant sets.
  • Nightly wheel verification now supports a separate source reference (ci_refspec) and a configurable publish subdirectory (PUBLISH_DIR, default verification), with updated Artifactory upload paths for release publishing.
  • Optional UCX SP-CX plugin support can be enabled for wheel builds.
• Documentation
  • Updated CI job inventory and wheel-build documentation to reflect the new release polling, gating, retry/partial-upload behavior, and publishing-folder rules.

[github-actions]

👋 Hi Alexey-Rivkin! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[Alexey-Rivkin]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-test-sanitizers · commit 9530ad79

TL;DR: The tsan "Test Sanitizer" stage was aborted (exit 143 / FlowInterruptedException) after the Jenkins build agent's remoting channel dropped (hudson.remoting.ChannelClosedException) mid-run — an infrastructure/agent-connectivity failure, not a NIXL code bug or a test hang. Retry the build on a healthy agent and/or stabilize the CI node; no source fix is warranted.

Full analysis

Summary: The x86_64/tsan "Test Sanitizer" stage (node 190/124) was killed with SIGTERM (exit code 143) while ucx_backend_test was still actively running under ThreadSanitizer.

Root cause: Loss of the Jenkins agent remoting channel, not a test failure or hang. At 07:48:41 the log records hudson.remoting.ChannelClosedException: Channel "...:nixl-ci-test-sanitizers-nixl-base-pytorch2606-cuda133-ubu-18cv2" ... The channel is closing down or has closed down. The workload continued printing Data verification: OK every ~8–15s with no multi-minute stall right up to 07:56:35, when Jenkins issued "Sending interrupt signal to process" → "Killing processes", terminated etcd/tar, and the step ended with exit code 143 and FlowInterruptedException. Steady output up to the kill rules out a hang; the abort was driven by the agent going unreachable. (The TSAN run is legitimately slow — ~40 min — but it was progressing.)

Implicated commit: unknown — no application/source commit is implicated; this is a CI-agent/infrastructure fault. The tested commit 9530ad7 (PR #1912) is not the cause.

File: N/A (Jenkins agent nixl-ci-test-sanitizers-nixl-base-pytorch2606-cuda133-ubu-18cv2, not a repo source file)

Suggested fix: Re-run the nixl-ci-test-sanitizers build; the failure is very likely transient. If it recurs, investigate the agent node health (OOM-killer, disk pressure, network/JNLP disconnects, or the container being evicted) around 07:48–07:56 — check dmesg/kernel OOM logs and the Jenkins agent connection logs on ...ubu-18cv2. Since TSAN of ucx_backend_test legitimately takes ~40 min and pushes agent resources, consider giving this stage a dedicated/larger executor or reducing its iteration count so it finishes before agent pressure builds. Do not raise the step time limit as a "fix" — the process was not hung and the kill was caused by the channel drop, not a timeout.

Related: none found.

[svc-nixl]

🤖 CI Triage Agent — AWS NIXL Validation · commit e090cc55

TL;DR: The AWS Batch test pod failed instantly because its git clone https://github.com/ai-dynamo/nixl could not authenticate (fatal: could not read Username for 'https://github.com'), so the job went to FAILED before any NIXL test ran. The clone inside the pod is unauthenticated and cannot access the repo/ref, so the workflow must pass credentials (or the PR ref) into the batch job.

Full analysis

Summary: "AWS NIXL Validation" job failed — the AWS Batch pod's git clone of the repository failed with a credential error, causing the batch job to reach terminal status FAILED after 12s.

Root cause: The command run inside the pod is git clone https://github.com/ai-dynamo/nixl && cd nixl && git checkout pull-request/1912 && ... (built in contrib/aws-efa/aws_test.sh:58). The clone is unauthenticated and got fatal: could not read Username for 'https://github.com': No such device or address (GHA log 11:16:34), immediately followed by fatal: expected flush after ref listing. Git dropped to an interactive username prompt with no TTY, meaning anonymous HTTPS access to the repo/ref was refused. Note also that even a successful anonymous clone could not git checkout pull-request/1912, since that PR ref only exists in the CI environment, not on the public remote — the clone step never got that far but the checkout would also fail. This is a hard, immediate failure (pod up 11:16:33 → clone fail 11:16:34 → FAILED 11:17:39, no large timestamp gaps), not a hang or timeout.

Implicated commit: Not a code-defect commit; the clone mechanism dates to 2709adcd (ovidiusm, "Unify AWS test execution", #1434) and the AWS_CMD construction persists there. The failure is environmental (auth/ref availability), triggered under PR #1912's context.

File: contrib/aws-efa/aws_test.sh:58

Suggested fix: Make the in-pod checkout able to fetch the PR ref with credentials, rather than an anonymous clone of a branch name that doesn't exist on the public remote. Concretely: (1) inject a token into the clone URL / git credential helper for the batch job (e.g. use an authenticated ${GITHUB_SERVER_URL} with a CI token), and (2) fetch the exact commit ref instead of the PR branch name — e.g. git clone <url> && cd nixl && git fetch origin ${GITHUB_SHA} && git checkout ${GITHUB_SHA} (using e090cc55… / the PR merge ref) so it doesn't depend on a pull-request/1912 branch existing on the remote. Also consider failing fast with a clear message when the clone returns no credentials, since the current flow only surfaces this via the generic "Failure running NIXL tests".

Related: none found (search the repo for prior "could not read Username" AWS batch clone failures; none surfaced in history).

[Alexey-Rivkin]

Tested on main branch with the test commit e090cc5 (reverted).
Looks good:

• https://nbuprod.blsm.nvidia.com/nbu-swx-nixl-main/job/nixl-ci-build-wheel-nightly/273/
• https://nbuprod.blsm.nvidia.com/nbu-swx-nixl-main/job/nixl-ci-build-wheel-nightly/274/

[svc-nixl]

🤖 CI Triage Agent — AWS NIXL Validation · commit 37c88a5e

TL;DR: The AWS Batch test pod aborted during .gitlab/build.sh when the nvm installer's git clone https://github.com/nvm-sh/nvm failed with could not read Username for 'https://github.com' (a transient GitHub HTTPS/auth hiccup); under set -e that killed the whole build, so the Batch job went to FAILED. Retry the job, and harden the nvm install step with retries / anonymous-clone config so a transient GitHub blip doesn't fail the build.

Full analysis

Summary: "AWS NIXL Validation" failed because the in-pod build script aborted while installing nvm; the AWS Batch job then reported FAILED.

Root cause: In the streamed pod log the last activity is the nvm installer: Cloning into '/root/.nvm'... fatal: could not read Username for 'https://github.com': No such device or address → Failed to clone nvm repo. install_nvm.sh does a git clone https://github.com/nvm-sh/nvm, which intermittently prompts for credentials / fails to complete (a transient GitHub-side issue — note rustup, uv, and the raw.githubusercontent.com download of install.sh in the same run all succeeded, so it's not a general network outage). Because .gitlab/build.sh runs under set -e and the nvm step (lines 223–225) has no retry or failure guard, the non-zero exit aborts the entire build. The pod exits non-zero, so the AWS Batch job transitions to FAILED (the outer poller reports "FAILED after 1s" only because kubectl logs -f had already returned by then). This is not caused by anything in PR #1912.

Implicated commit: unknown (the nvm install step predates recent history; the failure is an external/transient GitHub error, not a code regression)

File: .gitlab/build.sh:223-225 (nvm install step, under set -e at line 20)

Suggested fix: Re-run the job first — this is very likely a transient GitHub clone failure. To stop it recurring, make the nvm install robust: wrap it in a retry loop and/or set GIT_TERMINAL_PROMPT=0 so a credential prompt fails fast into a retry, e.g.:

export GIT_TERMINAL_PROMPT=0
for i in 1 2 3; do "${BUILD_TMP}"/install_nvm.sh && break || sleep 10; done

(Optionally pre-configure git config --global url."https://github.com/".insteadOf or use the tarball install method to avoid an interactive-credential path entirely.)

Related: none

[Alexey-Rivkin]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-test-sanitizers · commit 37c88a5e

TL;DR: The ASAN and TSAN sanitizer builds both failed during meson setup because the taskflow dependency is fetched via git clone from github.com, which the build container can't reach ("could not read Username for 'https://github.com'"). Fix by converting subprojects/taskflow.wrap from [wrap-git] to a [wrap-file] tarball download (as liburing.wrap already does).

Full analysis

Summary: meson setup fails in both sanitizer variants at the taskflow subproject fallback with a git clone/network error.

Root cause: subprojects/taskflow.wrap is a [wrap-git] wrap that runs git clone --depth 1 --branch v3.10.0 https://github.com/taskflow/taskflow.git. The sanitizer build containers have no GitHub egress/credentials, so git fails with fatal: could not read Username for 'https://github.com': No such device or address, aborting meson (meson.build:205). Other deps succeed: liburing is a [wrap-file] tarball and Abseil resolves from the system, so taskflow is the only dependency still relying on a live git clone.

Implicated commit: unknown (the failure is environmental — no GitHub access in the sandbox — not a specific code change; branch ci/release-wheel-pipeline likely runs these builds in a network-restricted context).

File: subprojects/taskflow.wrap:1-4 (git wrap); consumed at meson.build:205

Suggested fix: Convert taskflow.wrap to a [wrap-file] tarball wrap mirroring liburing.wrap, e.g.:

[wrap-file]
directory = taskflow-3.10.0
source_url = https://github.com/taskflow/taskflow/archive/refs/tags/v3.10.0.tar.gz
source_filename = taskflow-3.10.0.tar.gz
source_hash = <sha256 of the tarball>
source_fallback_url = <internal/wrapdb mirror>
patch_directory = taskflow

[provide]
taskflow = taskflow_dep

This uses an HTTP download (reachable via the CI package mirror) instead of a raw git clone. Alternatively, pre-install taskflow into the base image like Abseil, or configure git/network egress (or a git URL rewrite to an internal mirror) in the sanitizer containers.

Related: none found.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-test-sanitizers · commit 37c88a5e

TL;DR: The sanitizer builds (asan+ubsan and tsan) both failed at meson setup because the taskflow subproject is fetched via a live git clone from github.com, which fails in the CI container with could not read Username for 'https://github.com'; convert the taskflow wrap to a wrap-file tarball download (like liburing) or provide taskflow via the pre-installed environment.

Full analysis

Summary: Both parallel sanitizer Build stages (nodes 125 asan/ubsan and 124 tsan) failed during meson setup when resolving the taskflow dependency.

Root cause: subprojects/taskflow.wrap is a [wrap-git] entry pointing at https://github.com/taskflow/taskflow.git. During meson setup, Meson runs git clone --depth 1 --branch v3.10.0 https://github.com/taskflow/taskflow.git, which fails with fatal: could not read Username for 'https://github.com': No such device or address — the sanitizer container has no interactive terminal or credentials for github.com git access. Note liburing, which uses a [wrap-file] HTTPS tarball download with a wrapdb fallback URL, resolved successfully in the same run; only the git-clone-based taskflow dependency fails. This is an environment/dependency-fetch defect, not a code compile error.

Implicated commit: unknown (the taskflow wrap-git dependency; not attributable to any of the recent meson.build commits — the failure is the fetch method, subprojects/taskflow.wrap)

File: subprojects/taskflow.wrap:1-3 (the [wrap-git] / url = https://github.com/taskflow/taskflow.git), consumed at meson.build:205

Suggested fix: Replace the [wrap-git] taskflow entry with a [wrap-file] tarball download mirroring the liburing approach, e.g.:

[wrap-file]
directory = taskflow-3.10.0
source_url = https://github.com/taskflow/taskflow/archive/refs/tags/v3.10.0.tar.gz
source_filename = taskflow-3.10.0.tar.gz
source_hash = <sha256>
source_fallback_url = https://github.com/mesonbuild/wrapdb/releases/download/...
patch_directory = taskflow
[provide]
taskflow = taskflow_dep

Alternatively, pre-install taskflow into the sanitizer base image (so dependency('taskflow') is found on the system and the fallback is never triggered), or configure the CI git environment with proper github.com access/mirror so wrap-git clones can succeed non-interactively.

Related: none (no matching issue/PR found)

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 37c88a5e

TL;DR: The Compiling NIXL Docker Image stage failed because Meson's taskflow subproject fallback tried to git clone https://github.com/taskflow/taskflow.git inside the build container, which has no network/credentialed access to github.com. The fix is to satisfy taskflow without hitting github (mirror the source via the internal Artifactory, provide it as a system/pre-vendored dependency, or use a wrap-redirect to the internal mirror).

Full analysis

Summary: meson setup aborted during .gitlab/build.sh when resolving the taskflow dependency because the container cannot reach github.com to clone the fallback subproject.

Root cause: UCX built and installed fine; the failure is in the NIXL Meson configure step. meson.build:205 declares taskflow_proj = dependency('taskflow', fallback: ['taskflow', 'taskflow_dep']). taskflow is not available as a system pkg-config/cmake package (Run-time dependency taskflow found: NO), so Meson executed the wrap fallback which runs git clone --depth 1 --branch v3.10.0 https://github.com/taskflow/taskflow.git. The container has no github access/credentials, so git failed: fatal: could not read Username for 'https://github.com': No such device or address → ERROR: Git command failed. The same environment limitation is visible just above where a Cloning into 'taskflow'... also failed and the liburing dependency had to be pulled by tarball rather than git. This is an environment/dependency-provisioning issue, not a code defect in the branch.

Implicated commit: unknown — meson.build's taskflow fallback predates the recent listed commits; the branch ci/release-wheel-pipeline (PR #1912) exercises a container/build path that lacks github egress. No single commit in the recent history introduced the github clone.

File: meson.build:205 (taskflow fallback) — with the actual failure in the Docker build step running .gitlab/build.sh.

Suggested fix: Provide taskflow without cloning from github inside the container. Options, in order of preference:

• Vendor/pre-install taskflow in the base image (it is header-only) or install a system package so dependency('taskflow', ...) resolves before the fallback.
• Point the taskflow .wrap (in subprojects/) at the internal Artifactory mirror (urm.nvidia.com) via source_url/source_filename instead of a git clone of github.com — the same pattern liburing uses (tarball download), which succeeded.
• Pre-populate subprojects/packagecache (or run meson subprojects download at image-build time when egress exists) so no clone happens at build time.

Related: none found in issue/PR search.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit 37c88a5e

TL;DR: All four "Build image" stages failed because git clone https://github.com/... inside the container build could not reach GitHub (could not read Username for 'https://github.com': No such device or address) — an egress/DNS network failure to github.com from the build hosts, not a code defect. Re-run the build; if it persists, route these clones through the NVIDIA-internal GitHub mirror.

Full analysis

Summary: The container image builds (nixl & nixlbench, both x86_64 and aarch64) failed during Dockerfile RUN git clone steps for third-party deps (abseil-cpp, grpc + its submodules).

Root cause: git clone/submodule fetches against https://github.com fail with fatal: could not read Username for 'https://github.com': No such device or address and expected flush after ref listing. That error means git could not establish the HTTPS connection to github.com (name resolution / egress blocked or rate-limited), so it fell through to prompting for credentials on a non-interactive TTY. The intermittent pattern within a single run — several submodules (bloaty, boringssl, googletest, protobuf, xds) clone successfully while others (abseil-cpp, benchmark, c-ares, re2, zlib, …) fail and then fail again on retry — confirms flaky/throttled network egress to GitHub from the build hosts rather than a repository or Dockerfile bug. No commit on this branch changed these clone steps.

Implicated commit: none — not a code regression. (PR #1912 / commit 37c88a5 is unrelated to the failure.)

File: Dockerfile RUN git clone steps — abseil-cpp step (STEP 29/69 / [4/4] STEP 12/51) and grpc step ([4/4] STEP 13/51) in the nixl/nixlbench container build context.

Suggested fix:

1. First action: retry the build — this is a transient network failure and may simply pass on re-run.
2. If it recurs, the durable fix is to stop cloning third-party deps directly from github.com during the image build. Follow the pattern already established in commit [REDACTED:Hex High Entropy String] ("build: clone libxml2 from the GitHub mirror") and 029a8544 (switch to the NVIDIA internal Ubuntu mirror): point the abseil-cpp and grpc clones (and grpc's --recurse-submodules) at the NVIDIA-internal GitHub mirror, or vendor/cache these deps so the build does not depend on live GitHub egress. Also verify DNS/proxy egress to github.com on the container-build agents.

Related: none found in issue/PR search directly; see prior mirror-migration commits [REDACTED:Hex High Entropy String] and 029a8544 for the recommended approach.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 75ad92b9-1049-4034-89e7-5b7442382ddd in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 37c88a5e

TL;DR: The elastic EP test hung for the full 5-minute timeout 300 window and was killed (exit 124) because InfiniBand on gb-nvl-118-compute09 never came up (ports in PORT_INIT/PORT_DOWN), so UCX's ucp_device_remote_mem_list_create looped forever on UCS_ERR_NOT_CONNECTED. This is a node/fabric health problem, not a code defect — re-run on a healthy node and/or fix the IB fabric.

Full analysis

Summary: "Run DL NIXL EP tests" stage failed with exit code 124 — the elastic EP test (no_expansion.json) was killed by timeout 300 after hanging in device-memory-list creation.

Root cause: All 4 ranks hung in createMemList's retry loop (src/plugins/ucx/mem_list.cpp:163-174), which spins while ucp_device_remote_mem_list_create returns UCS_ERR_NOT_CONNECTED. The UCX connections never established because the node's InfiniBand ports were not up: the log shows 6 IB device(s) were detected, but accelerated IB support was not found! and ibv_devinfo reports mlx5_0–3 as PORT_INIT, mlx5_4–5 as PORT_DOWN, all with sm_lid: 0/port_lid: 65535 (no subnet-manager LID). From 12:52:35 to 12:57:31 the only output was the repeating Still waiting to create device memory list ... retrying warning — a hang, not slow progress. This is a fabric/node-health issue on gb-nvl-118-compute09 (slurm job 2016833), not a regression in the changed code.

Implicated commit: Not a code regression. The warning/loop was added by ef5aad2 (Raul Akhmetshin, PR #1410); the underlying failure is infrastructure (IB ports down on the allocated node).

File: src/plugins/ucx/mem_list.cpp:163-174 (the unbounded UCS_ERR_NOT_CONNECTED retry loop that manifests the hang)

Suggested fix: Infra: check/fix the InfiniBand fabric on gb-nvl-118-compute09 (subnet manager / port state — ports are stuck in PORT_INIT/PORT_DOWN) or drain the node and re-run the job on a host with ACTIVE IB ports. Code hardening (optional, to fail fast instead of hanging the whole timeout 300): give the while loop in createMemList an overall deadline so it throws a clear "peer not connected / IB down" error rather than spinning until the outer test timeout.

Related: PR #1410 (added the timeout warning), PR #1409/#1912 context branch ci/release-wheel-pipeline.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 37c88a5e

TL;DR: The Compiling NIXL EP Docker Image for DL stage failed because git clone https://github.com/uccl-project/uccl.git inside the podman build couldn't reach/authenticate to github.com (fatal: could not read Username for 'https://github.com': No such device or address). This is an environment/network access problem, not a code bug — the UCCL clone should be routed through the internal mirror or the build should be given github credentials.

Full analysis

Summary: Docker image build (stage 153) failed at exit code 128 when .gitlab/build.sh tried to git clone the UCCL repo from github.com.

Root cause: In .gitlab/build.sh, the UCCL clone (line 395) runs unconditionally when a GPU is present and PRE_INSTALLED_UCX_ENV is unset. In build #971 the log shows PRE_INSTALLED_ENV is set, skipping package installation (so all the other github.com clones — abseil, grpc, etc. — were skipped) but the PRE_INSTALLED_UCX_ENV build-arg was not passed (log: missing "PRE_INSTALLED_UCX_ENV" build argument). So the UCX/UCCL block executed and git clone https://github.com/uccl-project/uccl.git failed with fatal: could not read Username for 'https://github.com': No such device or address. Git prompted for credentials because github.com was unreachable/unauthenticated from inside the podman build --network host environment — a transient/network access failure, not a source defect. The apt traffic in the same step succeeded because it uses the internal urm.nvidia.com mirror; the raw github.com clone has no such mirror/credential path.

Implicated commit: Not a code regression — the failing clone predates recent changes. The surrounding EP build path was last touched by c984ce6 (lishapira, "CI: add vLLM + NIXL EP test to the EP CI job", #2154), but the failure is environmental.

File: .gitlab/build.sh:395 (git clone https://github.com/uccl-project/uccl.git)

Suggested fix: Restore/verify outbound access to github.com from the build node (or the git credential helper / token used for https://github.com). Retry the build first — this is a classic transient network/auth failure. If it recurs, either (a) pass --build-arg GIT_.../mount credentials so the clone can authenticate, or (b) mirror UCCL (and the other github.com deps) through the internal artifactory the way the Ubuntu packages already are, so the build no longer depends on direct github.com reachability. Do not treat this as a code change in NIXL itself.

Related: none (search returned only unrelated PR #2135).

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit 37c88a5e

TL;DR: The container Build image stages failed because git clone/submodule fetches from https://github.com intermittently returned "could not read Username for 'https://github.com': No such device or address" — a transient GitHub connectivity/rate-limit failure inside the Docker build, not a code defect. Retry the build or add resilient retries/mirroring for the git clones.

Full analysis

Summary: All four parallel container builds (nixl & nixlbench, x86_64 & aarch64) failed in their Build image stage during a RUN git clone ... step pulling third-party deps from github.com.

Root cause: Intermittent GitHub access failure during in-Dockerfile git clone. On x86_64 nixl (node 218) the etcd-cpp-apiv3 clone failed: fatal: could not read Username for 'https://github.com': No such device or address; on nixlbench x86_64 (node 233) the aws-sdk-cpp --recurse-submodules clone of aws-crt-cpp failed the same way, twice, after its own retry. In the same builds, wget of the libfabric release tarball and other clones succeeded, so this is transient network/rate-limit flakiness reaching github.com (git treats the failed anonymous fetch of a public repo as an auth prompt), not a missing credential or bad URL. Timestamps show the failures occurred immediately at the clone step (no multi-minute stalls), so this is not a hang or wall-clock timeout.

Implicated commit: unknown — not caused by a source change; the failing steps are pre-existing Dockerfile dependency clones. (PR #1912, branch ci/release-wheel-pipeline, only touches CI wiring.)

File: Dockerfile build steps in the container recipe — the RUN git clone --depth 1 https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3.git ... step and the RUN git clone --recurse-submodules ... https://github.com/aws/aws-sdk-cpp.git ... step (in contrib/).

Suggested fix: Re-run the build first — it is likely to pass. To make it robust, harden the git clones the way the tarball downloads already are: wrap them in a retry loop (e.g. for i in 1 2 3; do git clone ... && break || sleep 5; done), set git config --global http.lowSpeedLimit/http.lowSpeedTime and GIT_HTTP_MAX_REQUEST_BUFFER, and for submodule-heavy repos add git submodule update --init --recursive with retries rather than --recurse-submodules in one shot. Optionally pull dependencies through an internal mirror/cache to remove the github.com dependency entirely.

Related: PR #1912 (the PR under test); no existing issue tracks this specific flake.