fix(ci): switch test-e2e-sandbox to ubuntu-latest on self-hosted workflow by jyaunches · Pull Request #2294 · NVIDIA/NemoClaw

jyaunches · 2026-04-22T19:14:55Z

Summary

Switch test-e2e-sandbox from linux-amd64-cpu4 (NVIDIA self-hosted) to ubuntu-latest (GitHub-hosted) in the pr-self-hosted workflow. The snapshot rollback test fails 100% on the self-hosted runner but passes reliably on GitHub-hosted runners.

Related Issue

Unblocks all PRs gated on pr-self-hosted — the workflow has never had a passing run since it was added in #2121.

Changes

.github/workflows/pr-self-hosted.yaml: Change test-e2e-sandbox runs-on from linux-amd64-cpu4 to ubuntu-latest
Other E2E jobs (build-sandbox-images, test-e2e-gateway-isolation) remain on linux-amd64-cpu4 since they pass there

Diagnosis

The rollbackFromSnapshot() function in nemoclaw/src/blueprint/snapshot.ts does renameSync + cpSync on /sandbox/.openclaw (which contains symlinks to .openclaw-data). On the self-hosted runner's Docker environment, one of these operations throws — but the bare catch {} swallows the error and returns false, producing the "Rollback returned false" failure with no diagnostic output.

Confirmed in PR #2288 by temporarily switching the runner — test-e2e-sandbox passed immediately on ubuntu-latest.

Type of Change

Code change (feature, bug fix, or refactor)
Code change with doc updates
Doc only (prose changes, no code sample modifications)
Doc only (includes code sample changes)

Verification

npx prek run --all-files passes
npm test passes
Tests added or updated for new or changed behavior
No secrets, API keys, or credentials committed
Docs updated for user-facing behavior changes

AI Disclosure

AI-assisted — tool: Claude Code (pi agent)

Signed-off-by: Julie Yaunches jyaunches@nvidia.com

Summary by CodeRabbit

Chores
- Updated test environment configuration for improved compatibility and reliability of end-to-end snapshot tests.

…flow The snapshot rollback test (test 7 in e2e-test.sh) fails 100% on linux-amd64-cpu4 NVIDIA self-hosted runners but passes reliably on GitHub-hosted ubuntu-latest. The failure is in rollbackFromSnapshot() which silently catches an exception during renameSync/cpSync on the self-hosted runner's Docker storage driver. Confirmed by temporarily switching the runner in PR #2288 — test passed immediately on ubuntu-latest. The pr-self-hosted workflow has never had a passing run since it was added in #2121. The other E2E jobs (build-sandbox-images, test-e2e-gateway-isolation) remain on linux-amd64-cpu4 since they pass there. Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

coderabbitai · 2026-04-22T19:18:56Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 5cd7b825-5503-4067-991d-49b2222d8095

📥 Commits

Reviewing files that changed from the base of the PR and between d50452a and 0d8a70b.

📒 Files selected for processing (1)

.github/workflows/pr-self-hosted.yaml

📝 Walkthrough

Walkthrough

Modified the GitHub Actions workflow to change the e2e sandbox test runner from linux-amd64-cpu4 to ubuntu-latest, with added comments documenting Docker incompatibility issues on the self-hosted runner.

Changes

Cohort / File(s)	Summary
GitHub Actions Workflow `.github/workflows/pr-self-hosted.yaml`	Changed `test-e2e-sandbox` job runner from `linux-amd64-cpu4` to `ubuntu-latest` and added inline documentation explaining Docker storage/filesystem incompatibility on the original self-hosted runner.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

🐰✨ A runner switch, so swift and clean,
From CPU cores to cloud serene,
Docker plays nice on ubuntu's turf,
Where snapshots rollback with perfect surf!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly and accurately summarizes the main change: switching the test-e2e-sandbox job runner from a self-hosted runner to ubuntu-latest in the CI workflow.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/self-hosted-sandbox-runner

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

ericksoa

Confirmed this is the same rollback failure blocking all PRs on pr-self-hosted. The fix is minimal and targeted — only the affected job moves to ubuntu-latest, other E2E jobs stay on the self-hosted runner. test-e2e-sandbox already passed on this PR. LGTM.

## Summary Fix snapshot rollback failure on NVIDIA self-hosted runners and move all E2E jobs back to self-hosted. ## Root Cause NVIDIA self-hosted runners use Docker with the **containerd overlayfs snapshotter** (`io.containerd.snapshotter.v1`), while GitHub-hosted runners use the legacy **`overlay2`** driver. On the containerd snapshotter, directories can span different overlay layers, causing `rename(2)` to return **`EXDEV` (cross-device link not permitted)**. `rollbackFromSnapshot()` and `cutoverHost()` in `snapshot.ts` both used bare `renameSync()` which fails with EXDEV on these runners. The bare `catch {}` swallowed the error and returned `false`, producing the "Rollback returned false" failure with no diagnostic output. This also caused a secondary **`ERR_FS_CP_EINVAL`** — since the rename failed, `.openclaw` still existed when `cpSync` ran, and it detected it would be copying symlink targets into themselves. ## Diagnostic Evidence Confirmed via a matrix job across 4 runners: | Runner | Storage Driver | `renameSync` | `cpSync` | |--------|---------------|-------------|---------| | `linux-amd64-cpu4` | `overlayfs` (containerd) | ❌ EXDEV | ❌ ERR_FS_CP_EINVAL | | `linux-amd64-cpu8` | `overlayfs` (containerd) | ❌ EXDEV | ❌ ERR_FS_CP_EINVAL | | `linux-amd64-cpu16` | `overlayfs` (containerd) | ❌ EXDEV | ❌ ERR_FS_CP_EINVAL | | `ubuntu-latest` | `overlay2` (legacy) | ✅ | ✅ | ## Fix Add `moveSync()` helper that tries `renameSync` first (fast same-device path), then falls back to `cpSync` + `rmSync` on EXDEV. Both `cutoverHost()` and `rollbackFromSnapshot()` now use `moveSync()` instead of bare `renameSync()`. After the fix, all runners pass: | Runner | Before | After | |--------|--------|-------| | `linux-amd64-cpu4` | ❌ Rollback returned false | ✅ PASS | | `linux-amd64-cpu8` | ❌ Rollback returned false | ✅ PASS | | `linux-amd64-cpu16` | ❌ Rollback returned false | ✅ PASS | | `ubuntu-latest` | ✅ PASS | ✅ PASS | ## Changes ### `nemoclaw/src/blueprint/snapshot.ts` - Add `moveSync()` — cross-device-safe move with EXDEV fallback - `cutoverHost()` — use `moveSync` instead of `renameSync` - `rollbackFromSnapshot()` — use `moveSync` instead of `renameSync` (both the archive step and the recovery path) ### `nemoclaw/src/blueprint/snapshot.test.ts` - Add `moveSync` unit tests: same-device rename, EXDEV fallback (cpSync + rmSync), non-EXDEV re-throw - Add `rmSync` to the in-memory fs mock ### `.github/workflows/pr-self-hosted.yaml` - Move `test-e2e-sandbox` back to `linux-amd64-cpu4` (was on `ubuntu-latest` as a workaround since PR #2294) - Remove temporary diagnostic matrix job and `test/diag-container-fs.sh` ## Follow-up - [ ] Phase 4: Remove duplicate `sandbox-images-and-e2e` from `pr.yaml` (separate PR) ## Type of Change - [x] Code change (feature, bug fix, or refactor) ## Verification - [x] `moveSync` unit tests pass (22/22 snapshot tests green) - [x] `test-e2e-sandbox` passes on `linux-amd64-cpu4` (NVIDIA self-hosted) - [x] `test-e2e-gateway-isolation` passes on `linux-amd64-cpu4` - [x] All jobs pass on `ubuntu-latest` (no regression) - [x] No secrets, API keys, or credentials committed ## AI Disclosure - [x] AI-assisted — tool: Claude Code (pi agent) --- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com> --------- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

) ## Summary Fix snapshot rollback failure on NVIDIA self-hosted runners and move all E2E jobs back to self-hosted. ## Root Cause NVIDIA self-hosted runners use Docker with the **containerd overlayfs snapshotter** (`io.containerd.snapshotter.v1`), while GitHub-hosted runners use the legacy **`overlay2`** driver. On the containerd snapshotter, directories can span different overlay layers, causing `rename(2)` to return **`EXDEV` (cross-device link not permitted)**. `rollbackFromSnapshot()` and `cutoverHost()` in `snapshot.ts` both used bare `renameSync()` which fails with EXDEV on these runners. The bare `catch {}` swallowed the error and returned `false`, producing the "Rollback returned false" failure with no diagnostic output. This also caused a secondary **`ERR_FS_CP_EINVAL`** — since the rename failed, `.openclaw` still existed when `cpSync` ran, and it detected it would be copying symlink targets into themselves. ## Diagnostic Evidence Confirmed via a matrix job across 4 runners: | Runner | Storage Driver | `renameSync` | `cpSync` | |--------|---------------|-------------|---------| | `linux-amd64-cpu4` | `overlayfs` (containerd) | ❌ EXDEV | ❌ ERR_FS_CP_EINVAL | | `linux-amd64-cpu8` | `overlayfs` (containerd) | ❌ EXDEV | ❌ ERR_FS_CP_EINVAL | | `linux-amd64-cpu16` | `overlayfs` (containerd) | ❌ EXDEV | ❌ ERR_FS_CP_EINVAL | | `ubuntu-latest` | `overlay2` (legacy) | ✅ | ✅ | ## Fix Add `moveSync()` helper that tries `renameSync` first (fast same-device path), then falls back to `cpSync` + `rmSync` on EXDEV. Both `cutoverHost()` and `rollbackFromSnapshot()` now use `moveSync()` instead of bare `renameSync()`. After the fix, all runners pass: | Runner | Before | After | |--------|--------|-------| | `linux-amd64-cpu4` | ❌ Rollback returned false | ✅ PASS | | `linux-amd64-cpu8` | ❌ Rollback returned false | ✅ PASS | | `linux-amd64-cpu16` | ❌ Rollback returned false | ✅ PASS | | `ubuntu-latest` | ✅ PASS | ✅ PASS | ## Changes ### `nemoclaw/src/blueprint/snapshot.ts` - Add `moveSync()` — cross-device-safe move with EXDEV fallback - `cutoverHost()` — use `moveSync` instead of `renameSync` - `rollbackFromSnapshot()` — use `moveSync` instead of `renameSync` (both the archive step and the recovery path) ### `nemoclaw/src/blueprint/snapshot.test.ts` - Add `moveSync` unit tests: same-device rename, EXDEV fallback (cpSync + rmSync), non-EXDEV re-throw - Add `rmSync` to the in-memory fs mock ### `.github/workflows/pr-self-hosted.yaml` - Move `test-e2e-sandbox` back to `linux-amd64-cpu4` (was on `ubuntu-latest` as a workaround since PR NVIDIA#2294) - Remove temporary diagnostic matrix job and `test/diag-container-fs.sh` ## Follow-up - [ ] Phase 4: Remove duplicate `sandbox-images-and-e2e` from `pr.yaml` (separate PR) ## Type of Change - [x] Code change (feature, bug fix, or refactor) ## Verification - [x] `moveSync` unit tests pass (22/22 snapshot tests green) - [x] `test-e2e-sandbox` passes on `linux-amd64-cpu4` (NVIDIA self-hosted) - [x] `test-e2e-gateway-isolation` passes on `linux-amd64-cpu4` - [x] All jobs pass on `ubuntu-latest` (no regression) - [x] No secrets, API keys, or credentials committed ## AI Disclosure - [x] AI-assisted — tool: Claude Code (pi agent) --- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com> --------- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

jyaunches added the CI/CD label Apr 22, 2026

jyaunches self-assigned this Apr 22, 2026

ericksoa approved these changes Apr 22, 2026

View reviewed changes

cv approved these changes Apr 22, 2026

View reviewed changes

iamsh4 approved these changes Apr 22, 2026

View reviewed changes

brandonpelfrey approved these changes Apr 22, 2026

View reviewed changes

brandonpelfrey merged commit b8d64cf into main Apr 22, 2026
20 checks passed

This was referenced Apr 22, 2026

fix(onboard): classify expired API key as credential error #2132

Merged

feat(skills): add title tag normalization maintainer skill #2292

Merged

fix(ci): enable NVIDIA self-hosted runners for PR E2E tests #2374

Merged

wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions chore Build, CI, dependency, or tooling maintenance and removed CI/CD labels Jun 3, 2026

jyaunches deleted the fix/self-hosted-sandbox-runner branch June 12, 2026 13:52

Navigation Menu

fix(ci): switch test-e2e-sandbox to ubuntu-latest on self-hosted workflow - #2294

fix(ci): switch test-e2e-sandbox to ubuntu-latest on self-hosted workflow#2294
brandonpelfrey merged 1 commit into
mainfrom
fix/self-hosted-sandbox-runner

jyaunches commented Apr 22, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Apr 22, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

ericksoa left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

jyaunches commented Apr 22, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Diagnosis

Type of Change

Verification

AI Disclosure

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

ericksoa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jyaunches commented Apr 22, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Apr 22, 2026 •

edited

Loading