Repository navigation
fix(release): select AISim RC1 source and registry for 1.7.0 staging - #18232
Conversation
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> (cherry picked from commit f217642) Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> (cherry picked from commit 7a179b6) Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
❌ Dynamo PR CI failed — run 42097546370 (attempt 1) on
|
| Other | Jobs |
|---|---|
| deploy-operator | ❌ 1 |
Failure details
1 job failed: deploy-operator lost its connection to the vCluster API server while waiting for the operator Deployments, about 10 s into a 1800 s wait. This looks like CI infrastructure trouble, not a test failure.
❌ deploy-operator: vCluster API connection refused while waiting for grove-operator
Job: deploy-operator · Failed step: Setup vCluster and operator (sub-step "Wait for operator Deployments to be available") · Logs: gh run view --job 127496025128 -R ai-dynamo/dynamo --log-failed
##[group]Wait for operator Deployment rollouts (vCluster)
deployment.apps/dynamo-platform-dynamo-operator-controller-manager condition met
error: Get "https://127.0.0.1:8443/apis/apps/v1/namespaces/default/deployments/grove-operator": dial tcp 127.0.0.1:8443: connect: connection refused - error from a previous attempt: unexpected EOF
##[error]Operator Deployments did not become available within 1800s
##[error]Process completed with exit code 1.
[pod/ci-42097546370-0/syncer] ... grpc: addrConn.createTransport failed to connect to {Addr: "unix:///data/kine.sock"} ... "error reading server preface: read unix @->/data/kine.sock: use of closed network connection"
The Helm release was deployed, the CRDs were established, and the Dynamo operator controller-manager became Available. Then the kubectl wait for grove-operator failed because the vCluster API endpoint (127.0.0.1:8443) returned EOF and then refused connections. The step ended about 10 s into its 1800 s wait, so this was not a real timeout, even though the error message says "did not become available within 1800s". The vCluster syncer logs show repeated kine.sock connection errors, which suggests the vCluster API server was unstable. The grove-operator pod restarted once after exiting cleanly and was Running/Ready when the debug output was collected. This is a vCluster infrastructure signature; rerunning the failed jobs is likely enough.
For agents
{"pr": 18232, "run_id": 42097546370, "run_attempt": 1, "head_sha": "3105e42cb243e4a99ecb1d206475d22cd0cf0c6a",
"failures": [{"job": "deploy-operator", "job_id": 127496025128, "failed_step": "Setup vCluster and operator", "signature": "vCluster API 127.0.0.1:8443 connection refused (unexpected EOF) during kubectl wait for grove-operator", "tests": [], "log_cmd": "gh run view --job 127496025128 -R ai-dynamo/dynamo --log-failed"}]}Posted automatically by Devin for run 42097546370. Updated on every full-CI run of this PR.
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Overview
Allow Dynamo 1.7.0 crate staging to resolve the indexed AISim 0.14.0 RC1 package instead of asking crates.io for unpublished
aisim-core =0.14.0. Ordinary CI continues to build both AISim artifacts from Git commita420a6d8b26c767b250a145136f8732d75925069.Summary / Key Changes
aisim-core, version=0.14.0-rc.1, registry aliasaisim, and the matching source revision.aisimulate-coredependency alias to that registry/version and remove the matching Git patch. Advance the root, Python-binding, and KVBM Git patches and lockfiles to the RC1 source commit; Python wheels continue to build from that same pin.ARTIFACTORY_CARGO_DEPENDENCY_REGISTRIES; credentials use the existingARTIFACTORY_TOKENsecret. The source pin is checked before substitution.Approximately 200 lines of common staging logic, excluding tests/documentation/whitespace, plus release configuration and matching source pins. Merge #18231 first: the release workflow deliberately loads its staging script from the trusted main workflow commit. This release PR is prepared from that candidate commit; it is not yet a cherry-pick of a merged main change.
Registry configuration
In
ai-dynamo/dynamo, go to Settings → Secrets and variables → Actions → Variables and create the repository variableARTIFACTORY_CARGO_DEPENDENCY_REGISTRIESwith this exact JSON object:{"aisim":"sparse+https://artifactory.nvidia.com/artifactory/api/cargo/sw-dynamo-aisimulate-cargo-local/index/"}aisimmatches the registry alias inworkspace.metadata.artifactory.dependencies.aisim-core. The release manifest pins=0.14.0-rc.1; the variable supplies only the sparse index URL, including itssparse+prefix and trailing/index/.Keep credentials in the existing
ARTIFACTORY_TOKENsecret, with read access to the AISim Cargo repository and publication access to the Dynamo Cargo repository.ARTIFACTORY_CARGO_INDEXcontinues to select the Dynamo publication destination. Ensure theautomated-releaseenvironment does not override the repository variable with a different value.The repository variable is configured and verified, with no automated-release environment override. Main PR #18231 must merge before release PR #18232 can use this staging support.
The RC1 wheels are staged at
sw-dynamo-aisimulate-pypi-local/release/0.14.0/rc1/; Linux wheels come from successful AISim Full CI run 42130977533.Validation
aisim-core =0.14.0-rc.1viaaisim, removes only its Git patch, and preserves the original source-build manifest.eae825613615b5c1ba081fa467cbbfe149fa18cf90c6ec47d21e9ff5045edc3b.Issues and Ongoing Work
Refs OPS-9108.
Depends on #18231. The original release-only Git source pin landed in #18014; this follow-up advances the source to RC1 and supplies its publication counterpart. Release approval and merge remain with the release team.
Pull Request Checklist
/pr-checklistskill.