130 pipe by nv-nmailhot · Pull Request #1737 · ai-dynamo/nixl

nv-nmailhot · 2026-06-05T23:04:18Z

What?

Describe what this PR is doing.

Why?

Justification for the PR. If there is an existing issue/bug, please reference it. For
bug fixes, the 'Why?' and 'What?' can be merged into a single item.

How?

It is optional, but for complex PRs, please provide information about the design,
architecture, approach, etc.

…t-14 (drop GitLab)

dpressle · 2026-06-05T23:05:58Z

👀 Investigating stg-nixl smoke…

dpressle · 2026-06-05T23:06:33Z

🤖 CI Triage Agent — stg-nixl smoke · commit 907718d8

Summary: Unable to fetch logs — cannot diagnose without them.
Root cause: Unknown — log access failed.
Implicated commit: unknown
File: unknown
Suggested fix: Provide a direct link to the failed build log in a follow-up comment.
Log fetch attempts:

fetch_gha_logs({'run_id': 27044686736}) → [fetch_gha_logs ERROR] HTTP 404 for run 27044686736
fetch_jenkins_logs({'job': 'stg-nixl smoke', 'build': 27044686736}) → [fetch_jenkins_logs ERROR] HTTP 404 for stg-nixl smoke#27044686736
fetch_gha_logs({'run_id': 27044686736, 'job_id': 75373928170}) → [fetch_gha_logs ERROR] HTTP 404 for run 27044686736

dpressle · 2026-06-05T23:35:03Z

👀 Investigating stg-nixl smoke…

dpressle · 2026-06-05T23:35:39Z

🤖 CI Triage Agent — stg-nixl smoke · commit f852e142

I have all the evidence needed for a confident diagnosis. Here is the full report:

Summary: Both smoke-arm and smoke-amd jobs fail immediately at the aws-actions/amazon-ecr-login@v2 step with Not authorized to perform sts:AssumeRoleWithWebIdentity.

Root cause: The workflow uses OIDC-based AWS authentication (sts:AssumeRoleWithWebIdentity via the default AWS credential chain) to log in to Amazon ECR. This mechanism requires the id-token: write permission to be granted to the GitHub Actions job so that the runner can obtain a GitHub OIDC token and exchange it with AWS STS. That permission is absent or not propagated to these jobs.

This is a classic PR-from-fork / workflow-permissions problem: workflows triggered on PRs from forks (or workflows that do not explicitly declare permissions: id-token: write) receive a read-only or empty GITHUB_TOKEN scope and cannot generate the OIDC token needed to call sts:AssumeRoleWithWebIdentity. The error fires on the very first AWS step, in under 1 second — confirming it is a pure permission/config issue, not a code or logic bug in the PR itself (commit f852e14).

Key evidence from the log:

Secret source: Actions
Using default AWS credential chain.
##[error]Not authorized to perform sts:AssumeRoleWithWebIdentity

Both smoke-arm (aarch64 runner) and smoke-amd (x86_64 runner) hit the identical error at the identical step. No application code was ever executed.

Implicated commit: unknown — this is a workflow/infrastructure configuration issue, not a code regression in [REDACTED:Hex High Entropy String].

File: .github/workflows/stg-nixl-smoke.yml (workflow file not present in the checked-out source tree; likely lives only on the default branch and is being sourced from there)

Suggested fix: Add permissions: id-token: write (plus contents: read) to either the workflow-level or job-level permissions block in stg-nixl-smoke.yml. For example:

# At the workflow level (applies to all jobs):
permissions:
  id-token: write
  contents: read

# — OR at the individual job level (preferred for least-privilege):
jobs:
  smoke-arm:
    permissions:
      id-token: write
      contents: read
    ...
  smoke-amd:
    permissions:
      id-token: write
      contents: read
    ...

Additionally, verify that:

The AWS IAM role's trust policy allows the OIDC provider token.actions.githubusercontent.com and that the sub condition matches PRs from this repo (e.g. repo:ai-dynamo/nixl:pull_request).
If this workflow is triggered by pull_request events from forks, note that GitHub blocks id-token: write for fork PRs by default — the trigger may need to be changed to pull_request_target (with appropriate sandboxing) or the workflow must be approved/run from a trusted context.

Related: PR #1710 — Nmailhot/nixl infra (open, potentially related infrastructure/CI work)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 85aa290d-07d7-45ab-8e33-dc54098efb6c in the triage console for the audit trail.

COPY the DDN Infinia libs from the mirrored ECR image nixl/infinia-libs:v2.4.0-beta.1 (/infinia/${ARCH}/) into /opt/ddn/red so meson auto-detects red_client and compiles the INFINIA backend into the release wheel. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

A build ARG used in a FROM line must be declared in the global, pre-first-FROM scope; it was declared after the cuda stage so docker saw it as undeclared ("base name should not be blank"). Move the ARG to the top alongside BASE_IMAGE. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

INFINIA now detects/compiles, but libred_client.so pulls a dep closure the AlmaLinux-8 manylinux base lacks. Add them: - libuuid-devel (uuid/uuid.h, fixes the compile) - EPEL + jansson + libjose + xxhash-libs (libjansson.so.4, libjose.so.0, libxxhash.so.0) - source-build liburing 2.6 (EL8 only ships liburing.so.1; needs .so.2) - source-build OpenLDAP 2.6.8 (EL8 ships libldap-2.4.so.2; needs the 2.6 soname libldap.so.2 + liblber.so.2), against openssl3 - LD_LIBRARY_PATH/LIBRARY_PATH for /opt/ddn/red/lib + /usr/local/lib openssl3 (libssl/libcrypto.so.3) and libcurl.so.4 are already source-built. Remaining unknown: libprotobuf.so.32 (DDN's protobuf soname) may differ from nixl's gRPC protobuf — letting CI surface it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

libred_client/libred_async are DDN proprietary and already auditwheel --excluded in build-wheel.sh; the wheel ships only the plugin, which loads libred_* at runtime from a DDN install. Fix the stale comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

ovidiusm · 2026-06-16T12:29:02Z

-ARG BASE_IMAGE
-ARG BASE_IMAGE_TAG
-FROM ${BASE_IMAGE}:${BASE_IMAGE_TAG}
+# === Option B (DRAFT, UNVALIDATED) ===========================================


I would remove this comment (Option B / Unvalidated) if this is the final solution

ovidiusm · 2026-06-16T12:32:18Z

+            cuda_version: "12.9"
+            arch: x86_64
+            runner: amd
+          # Option B: manylinux jobs build on the public PyPA manylinux_2_28 base


Could we please clean up this comment (Option B)?

ovidiusm

LGTM, please see minor comments

ovidiusm · 2026-06-22T15:01:38Z

/build

ovidiusm · 2026-06-22T15:01:57Z

/build

Brings nixl's CI to main as a GitHub Actions pipeline, replacing the GitLab mirror+trigger flow: - .github/workflows/ci.yml — version + 5-way build matrix (build-nixl + manylinux x86/arm x cuda12.9/13), wheel security scan, and Artifactory wheel/crate upload. RC generation runs on a push to a release/** branch (i.e. a PR merged into release/<x.y.z> builds + uploads the RC), or a manual workflow_dispatch. Test/verify jobs are omitted for now (re-add once green). - contrib/Dockerfile.manylinux — Option B (public PyPA manylinux_2_28 + NGC CUDA, no GitLab base) plus the INFINIA libs stage and its build deps. - contrib/build-container.sh — --cuda-version + provenance/sbom flags. - meson.build — build_tests gate fix; .github/actionlint.yaml. Squashed from the 130-pipe work (#1737); the 1.3.0 version bump and INFINIA source are already on main (#1738). The GitLab nSpect/scan trigger comes in the follow-up release-pipeline change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Brings nixl's CI to main as a GitHub Actions pipeline, replacing the GitLab mirror+trigger flow: - .github/workflows/ci.yml — version + 5-way build matrix (build-nixl + manylinux x86/arm x cuda12.9/13), wheel security scan, and Artifactory wheel/crate upload. RC generation runs on a push to a release/** branch (i.e. a PR merged into release/<x.y.z> builds + uploads the RC), or a manual workflow_dispatch. Test/verify jobs are omitted for now (re-add once green). - contrib/Dockerfile.manylinux — Option B (public PyPA manylinux_2_28 + NGC CUDA, no GitLab base) plus the INFINIA libs stage and its build deps. - contrib/build-container.sh — --cuda-version + provenance/sbom flags. - meson.build — build_tests gate fix; .github/actionlint.yaml. Hardening (CodeRabbit review): add a concurrency group; fail the build job on empty output (if-no-files-found: error + no-wheels check) instead of masking it; pass the Artifactory token to the upload containers via env (drop set -x / no token in the traced command); persist-credentials: false on checkout; document --cuda-version in build-container.sh help. Squashed from the 130-pipe work (#1737); the 1.3.0 version bump and INFINIA source are already on main (#1738). The GitLab nSpect/scan trigger comes in the follow-up release-pipeline change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… (#1803) Brings nixl's CI to main as a GitHub Actions pipeline, replacing the GitLab mirror+trigger flow: - .github/workflows/ci.yml — version, 5-way build matrix (build-nixl + manylinux x86/arm × cuda12.9/13), scan, CPU tests, Artifactory wheel/crate upload, gated on tag/release. - contrib/Dockerfile.manylinux — Option B (public PyPA manylinux_2_28 + NGC CUDA, no GitLab base) plus the INFINIA libs stage and its build deps. - contrib/build-container.sh — --cuda-version + provenance/sbom flags. - meson.build — build_tests gate fix; .github/actionlint.yaml. Squashed from the 130-pipe work (#1737); the 1.3.0 version bump and INFINIA source are already on main (#1738), so this is the CI-pipeline delta only. The stg-nixl-* staging-validation workflows are included pending a decision to drop them; ci.yml still carries the TEMP ci-release-test label gate. ## What? _Describe what this PR is doing._ ## Why? _Justification for the PR. If there is an existing issue/bug, please reference it. For bug fixes, the 'Why?' and 'What?' can be merged into a single item._ ## How? _It is optional, but for complex PRs, please provide information about the design, architecture, approach, etc._  ## Summary by CodeRabbit * **Chores** * Migrated continuous integration from GitLab to GitHub Actions with PR/push/tag triggers, security scanning, and release-focused artifact publishing. * Added support for CUDA-version selection during image builds and expanded multi-variant (x86_64/ARM, CUDA variants, manylinux) build outputs. * Refreshed the manylinux container build to use public base images, improving toolchain/library readiness for wheel builds. * Adjusted build behavior so test binaries are included when building release test artifacts. * Added repository linting configuration to recognize known self-hosted runner labels.  Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…e / #1737) (#1803)" This reverts commit c8e7b78.

…e / ai-dynamo#1737) (ai-dynamo#1803)" This reverts commit c8e7b78.

nv-nmailhot added 12 commits June 5, 2026 16:03

test: smoke test infra on github

c6616cc

stg-nixl smoke: trigger on push for branch val

b88df03

fix wf

ba0a2b4

staging build test: compile wheels + crates on stg-nixl runner

926f473

update to add new builds

d5b0e6f

draft Option B: manylinux on public PyPA base + NGC CUDA + gcc-toolse…

aa55c16

…t-14 (drop GitLab)

Option B: CMAKE_POLICY_VERSION_MINIMUM=3.5 for CMake4 vs gRPC c-ares

551f948

fix test ci

13a02ed

TEMP: allow release path on PR via ci-release-test label

7c62fea

increase timeout

a55ee19

ci: trigger on PR labeled (release-test gate already present)

3cada53

fix upload path

907718d

nv-nmailhot requested review from a team as code owners June 5, 2026 23:04

pull-request-size Bot added the size/XL label Jun 5, 2026

copy-pr-bot Bot had a problem deploying to SWX_AWS June 5, 2026 23:04 Failure

nv-nmailhot added the ci-release-test label Jun 5, 2026

chore: Update release version for NIXL

f852e14

nv-nmailhot requested review from aranadive, brminich and ovidiusm as code owners June 5, 2026 23:34

copy-pr-bot Bot had a problem deploying to SWX_AWS June 5, 2026 23:34 Failure

nv-nmailhot temporarily deployed to release June 6, 2026 00:44 — with GitHub Actions Inactive

copy-pr-bot Bot had a problem deploying to SWX_AWS June 9, 2026 20:49 Failure

copy-pr-bot Bot had a problem deploying to SWX_AWS June 9, 2026 21:11 Failure

copy-pr-bot Bot had a problem deploying to SWX_AWS June 10, 2026 00:41 Failure

nv-nmailhot temporarily deployed to release June 10, 2026 01:59 — with GitHub Actions Inactive

copy-pr-bot Bot had a problem deploying to SWX_AWS June 10, 2026 05:36 Failure

nv-nmailhot temporarily deployed to release June 10, 2026 06:50 — with GitHub Actions Inactive

ovidiusm reviewed Jun 16, 2026

View reviewed changes

Comment thread .github/workflows/ci.yml

This was referenced Jun 16, 2026

Scripts to package wheels using Rocky Linux 8 (0.6.0) #1138

Closed

Scripts to package wheels using Rocky Linux (NIXL 1.0.0) #1137

Closed

ovidiusm approved these changes Jun 16, 2026

View reviewed changes

aranadive approved these changes Jun 16, 2026

View reviewed changes

Merge branch 'release/1.3.0' into 130-pipe

caa71a1

copy-pr-bot Bot had a problem deploying to SWX_AWS June 22, 2026 15:01 Failure

ovidiusm temporarily deployed to release June 22, 2026 16:19 — with GitHub Actions Inactive

nv-nmailhot mentioned this pull request Jun 22, 2026

ci: add native GitHub Actions pipeline (squashed from 130-pipe / #1737) #1803

Merged

dpressle pushed a commit that referenced this pull request Jun 24, 2026

Revert "ci: add native GitHub Actions pipeline (squashed from 130-pip…

5bd63eb

…e / #1737) (#1803)" This reverts commit c8e7b78.

NirWolfer pushed a commit to NirWolfer/nixl that referenced this pull request Jun 24, 2026

Revert "ci: add native GitHub Actions pipeline (squashed from 130-pip…

a9f456b

…e / ai-dynamo#1737) (ai-dynamo#1803)" This reverts commit c8e7b78.

nv-nmailhot closed this Jun 29, 2026

Navigation Menu

130 pipe - #1737

130 pipe#1737
nv-nmailhot wants to merge 18 commits into
release/1.3.0from
130-pipe

nv-nmailhot commented Jun 5, 2026

Uh oh!

dpressle commented Jun 5, 2026

Uh oh!

dpressle commented Jun 5, 2026

Uh oh!

dpressle commented Jun 5, 2026

Uh oh!

dpressle commented Jun 5, 2026

Uh oh!

ovidiusm Jun 16, 2026

Uh oh!

ovidiusm Jun 16, 2026

Uh oh!

Uh oh!

ovidiusm left a comment

Uh oh!

ovidiusm commented Jun 22, 2026

Uh oh!

ovidiusm commented Jun 22, 2026

Uh oh!

This branch had an error being deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

nv-nmailhot commented Jun 5, 2026

What?

Why?

How?

Uh oh!

dpressle commented Jun 5, 2026

Uh oh!

dpressle commented Jun 5, 2026

Uh oh!

dpressle commented Jun 5, 2026

Uh oh!

dpressle commented Jun 5, 2026

Uh oh!

ovidiusm Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

ovidiusm Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ovidiusm left a comment

Choose a reason for hiding this comment

Uh oh!

ovidiusm commented Jun 22, 2026

Uh oh!

ovidiusm commented Jun 22, 2026

Uh oh!

This branch had an error being deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants