Repository navigation
CI: retry git fetches and trim gRPC submodules in dependency builds - #2291
Alexey-Rivkin wants to merge 1 commit into
Conversation
|
👋 Hi Alexey-Rivkin! Thank you for contributing to ai-dynamo/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
|
/build |
44c466f to
e1fe3da
Compare
|
/build |
|
🤖 CI Triage Agent — TL;DR: The Docker build for Full analysisSummary: Stage "Build Docker" (node 55) failed: Root cause: Build-log evidence (14:41:31Z, stage 55):
Implicated commit: File: Suggested fix: Make the retry helper self-contained rather than path-dependent. Preferred: move the loop into retry() {
for i in 1 2 3 4 5; do
"$@" && return 0
[ "$i" -lt 5 ] && sleep $((i * 15))
done
return 1
}and delete line 25 of Related: PR #2291 (#2291) — the PR under test; this is a self-inflicted failure in that PR, not a pre-existing break.
|
|
🤖 CI Triage Agent — TL;DR: The sanitizer Docker build fails at Full analysisSummary: Stage "Build Docker" (both parallel variants, nodes 55 and 65) failed: Root cause: Deterministic missing-file error, not a hang or flake — the log shows continuous activity right up to the failure (libfabric built and installed at 14:42:18, failure 0.3 s later). The last lines are: Commit e1fe3da added Implicated commit: [REDACTED:Hex High Entropy String] — Alexey Rivkin, "CI: retry git fetches and trim gRPC submodules in dependency builds" File: Suggested fix: Pick one, in order of preference:
Related: PR #2291 (branch
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage "Build Docker" ( Root cause: Missing file inside the build context of the image, not a network/flake issue. The log shows:
Implicated commit: File: Suggested fix: Either
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Root cause: Not infra, not a timeout — a missing-file error in the new retry plumbing. The build image ran Everything before it (libfabric 2.7.0 download/configure/make/install, azurite npm install) completed normally, and log timestamps are continuous — no hang, no node problem. Note that Implicated commit: [REDACTED:Hex High Entropy String] — "CI: retry git fetches and trim gRPC submodules in dependency builds", Alexey Rivkin (adds both File: Suggested fix: Make the helper independent of the repo layout inside the image. Two changes, either of which resolves it (do both for robustness):
Also worth grepping the other CI Dockerfiles/scripts touched by e1fe3da for the same Related: PR #2291 (branch
|
e1fe3da to
9b0d412
Compare
|
/build |
|
🤖 CI Triage Agent — TL;DR: Every parallel branch of build #800 failed because Full analysisSummary: All four parallel branches failed: "Prepare environment" with Root cause: Commit 9b0d412 ("CI: retry git fetches and trim gRPC submodules in dependency builds", branch Implicated commit: 9b0d412 — Alexey Rivkin, 2026-09-28T14:23:40Z File: Suggested fix: |
9b0d412 to
ca8e827
Compare
|
/build |
ca8e827 to
8e74f59
Compare
|
/build |
|
🤖 CI Triage Agent — TL;DR: PR #2291's new Full analysisSummary: All four parallel "Build image" stages of Root cause:
Implicated commit: [REDACTED:Hex High Entropy String] — Alexey Rivkin, "CI: retry git fetches and trim gRPC submodules in dependency builds" File: contrib/Dockerfile:161 ( Suggested fix:
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage "Build Docker" (node 63) failed at the Root cause: The Dockerfile step leaves Everything before this (Mooncake, azure-sdk-for-cpp, gtest-parallel, azure-cli) built and installed cleanly, and the log shows continuous activity with no stalls — this is a hard failure, not a timeout or an infra problem. It is also unrelated to the PR under test (#2291, Implicated commit: File: Suggested fix:
Related: PR #2292 — CI: pin the vLLM release for the elastic EP test (direct fix); PR #2154 — CI: add vLLM + NIXL EP test to the EP CI job (origin of the patch)
|
The container and test builds fetch about 250 repositories anonymously from github.com per PR run. During busy hours a small fraction of those requests is refused (git reports "could not read Username"), which since mid-September failed roughly 60% of nixl-ci-build-container-pr runs and a third of nixl-ci-non-gpu runs. Two levers, both without changing what gets built: - Retry every git fetch up to 5 times with growing backoff (helper script in the images, shell function in build.sh, inline loop for the host-side ucx-src clone), pre-fetch the wrap-git meson subprojects the same way, and run the UCX submodule init under retry before autogen.sh, whose own gpunetio fetch failure is swallowed and only surfaces later in make. - Initialize only the gRPC submodules this cmake configuration consumes: protobuf, c-ares, re2 and the xds proto set. SSL, Abseil and zlib come from the prefix, tests and the OpenTelemetry plugin are off, so their submodules (including opentelemetry-cpp's nested tree) were fetched for nothing. aws-sdk-cpp keeps its full CRT tree, but through a retried submodule init instead of --recurse-submodules.
8e74f59 to
9dee74d
Compare
|
/build |
|
🤖 CI Triage Agent — TL;DR: The DL GPU image build died at Full analysisSummary: Stage "Compiling NIXL Docker Image for DL" (build #2419) failed at Root cause: Commit 8e74f59 added From Implicated commit: [REDACTED:Hex High Entropy String] — Alexey Rivkin, "CI: retry git fetches and trim gRPC submodules in dependency builds" File: Suggested fix: Resolve the script directory to an absolute path once, before any SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
. "${SCRIPT_DIR}/../.ci/scripts/common.sh"
retry() { "${SCRIPT_DIR}/../.ci/scripts/retry.sh" "$@"; }
Related: #2291
|
|
🤖 CI Triage Agent — TL;DR: The "Build Docker" stage failed because Full analysisSummary: Root cause: Not a code defect in the PR under test ( The log shows it picked up Because the reference floats to whatever vLLM tagged most recently, upstream edits around Implicated commit: No nixl commit broke this — the trigger is upstream vLLM's v0.30.0 release. The fragile mechanism was introduced in File: The Suggested fix: Pin Related: #2292 (CI: pin the vLLM release for the elastic EP test — the direct fix); #2154 (introduced the patch + floating |
Makes `.gitlab/build.sh` fail at the Mooncake clone, with git's error,
when that clone fails. Today the build carries on and fails later with a
misleading exit 127.
The `sed -i '/liburing-dev/d' dependencies.sh` line in the Mooncake
block has no trailing `&& \`, so the subshell is split into two `&&`
lists. `set -e` does not fire on a failure inside a list. A failed clone
therefore falls through to `bash dependencies.sh` in `${BUILD_TMP}`,
which fails with `No such file or directory` and exit 127. This happened
in nixl-ci-gpu #3885, where the clone failed with `curl 56 Recv failure:
Connection reset by peer`.
Running the block on its own with `git` stubbed out confirms it: a
failed clone exits 127 before the change and exits 128 at the clone
after it. A successful clone behaves the same in both cases.
Changes:
- `.gitlab/build.sh` — add the missing `&& \` after the `sed` line in
the Mooncake block.
#2291 adds `retry` to this same clone, so this PR doesn't add a retry.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* The build now proceeds to subsequent steps only when dependency
installation succeeds. If installation fails, later steps are skipped,
helping ensure the reported build result reflects whether its required
dependencies were installed successfully. This change affects the build
process and does not alter end-user features or application behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: NirWolfer <nwolfer@nvidia.com>
Select EP proxy mode through backend parameters while retaining direct mode as the default. Derive topology from EP rank capacity and proxy settings, erase the inherited num_workers default, and disable shared background progress because the proxy owns its progress threads. Signed-off-by: Tomer Davidor <tdavidor@nvidia.com>
## What? Two follow-ups to #2241 (generic S3-over-RDMA in the OBJ plugin): 1. Keep cuObject/RDMA headers out of `s3_accel/client.h` to fix a wheel-build OOM. 2. Review cleanups that came in after #2241 merged. ## Why? - `s3_accel/client.h` included `object/rdma/rdma.h`, so every file including it compiled the full cuObject header set. That pushed compiler memory high enough to OOM the wheel build. - Review cleanups: - The cuObject accelerated tests skipped on any construction failure, which hid real errors. ## How? - `client.h` forward-declares the RDMA types; `rdma.h` is included only in the `.cpp` files that need it. - Empty accel `type` → `"s3"` is now resolved in the engine registry instead of `createAccelEngine`. - cuObject accelerated tests skip when `NIXL_OBJ_ENDPOINT_OVERRIDE` is unset (like the Dell tests) and only skip a construction failure when the fast path is unavailable; anything else fails. - `const nixl_b_params_t *` on the param getters, plus comment fixes. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Object acceleration requests without an explicitly specified engine now use the S3 engine when available. If it is unavailable, the request reports that no engine type was specified. * **Bug Fixes** * Zero-byte RDMA writes are now treated as successful, preventing unnecessary retries. * Accelerated-object tests now skip when the endpoint is unset or the fast path is unavailable, while reporting other engine construction failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Nothing verified the closed-source `ucx-spcx-plugin` still compiles when its ref or the UCX under it moved — that only surfaced in the nightly. Building it in the per-PR wheel job was not an option: this repo is public, everything in a PR's checkout is editable by whoever opened the PR, and that job's console can be published back to the PR, so cloning the plugin here would hand its read credential and its source to PR-authored code. `nixl-ci-build-wheel` now triggers `nixl-ci-compile-ucx-plugin` from `pipeline_start` and fails the job if it fails, before any wheel is built. That job's definition, matrix and build scripts live in the plugin repo (`ucx-spcx-plugin!45`), so a PR can choose which ref to build but cannot change what the job does; it runs on the NIXL Jenkins only so it can stage the built module on the shared PVC. Both arches build in one run, and only the module and a pass/fail cross back. - `build-wheel-matrix.yaml`: `pipeline_start` triggers the compile job and errors on failure; adds `UCX_SPCX_PLUGIN_REF` and `SPCX_STAGING_ROOT`; Build Wheel passes `--ucx-spcx-plugin-install` - `build-container.sh`: new `--ucx-spcx-plugin-install <dir>` stages a pre-built module into the build context, needing no gitlab credentials - `Dockerfile.manylinux`: installs the staged module when present, otherwise keeps compiling from staged source — `nixl-ci-build-wheel-nightly` does a full two-stage build and has no published `wheel_base` to compile against yet - `.gitignore`, `.ci/docs/ci-overview.md`, `.ci/docs/build-wheel-matrix-ci.md` updated Blocked on `ucx-spcx-plugin!45` merging to `v0.3.x` and DevOps applying `proj_jjb_nixl.yaml` to the NIXL Jenkins. The top commit pins the plugin ref to the MR branch so the flow can be exercised before then, and must be reverted before merge. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Wheel and container builds can install pre-built UCX SPCX plugin modules. - Nightly CUDA 12 and 13 wheel builds use reusable, architecture-specific base images. - **Bug Fixes** - Builds now fail when plugin compilation fails or required staged plugin files are missing. - Plugin staging directories are validated before use. - **Documentation** - Updated CI guidance to describe plugin workflows, sourcing, and security boundaries. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: NirWolfer <nwolfer@nvidia.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…ms (#2386) When the AWS EFA Batch job fails, the wrapper prints only `Job reached terminal status FAILED after <N>s`. That makes a timeout, a lost node and a real test failure look the same. Three AWS failures on 2026-10-07 show the problem: - PR #2211: etcd received SIGTERM at 19:34:45.97, exactly 60 minutes after `timeout 60m bash contrib/aws-efa/aws_test_remote.sh` started at 18:34:46.03. GNU `timeout` signals the whole process group, so this was the in-pod `TEST_TIMEOUT`, not an external kill. - PR #2328: `kubectl logs -f` ended with `http2: server sent GOAWAY`, and the `--previous` fallback failed with `previous terminated container "default" ... not found` because the container was still running. The job went FAILED 60 minutes 11 seconds after `timeout 60m` started, so this was also the in-pod timeout. Both runs were slowed by that evening's apt slowdown (build about 53 minutes, against about 22 normally). - PR #2359: the log ended cleanly 7 minutes into the gRPC build, and Batch marked the job FAILED 13 minutes later. The cause is not in the log. In the AWS run logs I sampled, the GOAWAY drop hit 4 of about 67 runs, three of them otherwise successful. The `--previous` fallback never recovered any output. Changes: - `contrib/aws-efa/aws_test.sh`: a new `report_job_failure` runs when the job fails to start or does not reach `SUCCEEDED`. It prints the Batch `statusReason` and, for each attempt (`eksAttempts`), the pod, node, status reason and each container's exit code and reason. If an exit code is 124, it prints that the in-pod test timeout expired. - `contrib/aws-efa/aws_test.sh`: a new `follow_pod_logs` replaces `kubectl logs -f || kubectl logs --previous`. It follows the log with `--timestamps` and prints each line without its timestamp. If the stream ends while the container is still running, it reconnects with `--since-time` set to the last line received, up to 10 times. kubelet applies `--since-time` with 1 s granularity and prints timestamps at a fixed nanosecond width, so lines up to the last one received are skipped by comparing timestamps. - `contrib/aws-efa/README.md`: describe both behaviours in the execution flow. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * AWS test runs now stream timestamped pod logs and retry up to 10 times if the stream drops while a container is running, resuming after the last received timestamp without repeating lines. * Failure reports now include AWS Batch status and status reason, plus each attempt’s pod, node, and container exit details. When `TEST_TIMEOUT` is set and a container exits with code 124, reports identify an in-pod timeout. * **Documentation** * Updated test execution guidance to describe log reconnections and the additional failure details. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: NirWolfer <nwolfer@nvidia.com>
This branch has not been deployed
2 participants
What?
Retry git fetches in the dependency builds and fetch only the gRPC submodules the build uses.
Why?
could not read Username), failing ~60% ofnixl-ci-build-container-prruns and a third ofnixl-ci-non-gpuruns.How?
.ci/scripts/retry.sh(5 attempts, backoff), copied into both images and used by.gitlab/build.shand the container matrices, around everygit clone/fetch/submodule update.retry git submodule update --initbeforeautogen.sh(its own gpunetio fetch failure is swallowed and only fails later in make).package, tests and OTel plugin off). ~26 -> 8 clones per gRPC build.meson subprojects download taskflow prometheus-cppunderretrybeforemeson setup.Summary by CodeRabbit