Skip to content

test(e2e): add Brev launchable install-flow smoke test #2677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jyaunches merged 5 commits into from
Apr 29, 2026
Merged

test(e2e): add Brev launchable install-flow smoke test #2677

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
04edd8d
test(e2e): add Brev launchable install-flow smoke test (#2599)
jyaunches Apr 28, 2026
16490de
refactor(ci): rename e2e-brev → e2e-branch-validation for clarity
jyaunches Apr 29, 2026
1188018
fix: merge main — resolve nightly-e2e.yaml conflict, add dispatch guard
jyaunches Apr 29, 2026
ceff955
fix(e2e): suppress shellcheck SC2024 in launchable smoke test
jyaunches Apr 29, 2026
becf6b3
fix(e2e): move shellcheck disable to its own line
jyaunches Apr 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 27 additions & 8 deletions .github/workflows/e2e-brev.yaml → .github/workflows/e2e-branch-validation.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,30 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

name: e2e-brev
name: e2e-branch-validation

# Ephemeral Brev E2E: provisions a cloud instance, bootstraps NemoClaw,
# runs test suites remotely, then tears down. Use workflow_dispatch to
# trigger manually from the Actions tab, or workflow_call from other workflows.
# ─── Branch Validation E2E ───────────────────────────────────────────────────
#
# PURPOSE: Validates a specific git branch by installing NemoClaw FROM SOURCE
# on an ephemeral Brev cloud instance. Answers the question: "Does this branch
# work if you install from source on a clean machine?"
#
# HOW IT WORKS:
# 1. Provisions a fresh Brev CPU instance (~4 vCPU, 16 GB RAM)
# 2. Rsyncs the checked-out branch code to the VM
# 3. Runs install.sh from source + onboards a sandbox
# 4. Executes the selected test suite against the live sandbox
# 5. Tears down the instance (unless keep_alive=true)
#
# WHEN TO USE:
# - Before merging a PR that touches onboard, sandbox, security, or infra
# - To validate a branch works end-to-end on a clean Linux environment
# - To run security regression suites (credential-sanitization, injection)
# - Manually via workflow_dispatch, or called from other workflows
#
# NOTE: This does NOT test the community install path (launch-plugin.sh).
# For validating what Brev Launchable users actually get, see the
# e2e-launchable-smoke job in nightly-e2e.yaml.
Comment on lines +25 to +27

@coderabbitai coderabbitai Bot Apr 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Fix stale launchable job identifier in the header note.

The comment references e2e-launchable-smoke, but this PR context uses launchable-smoke-e2e. Keeping this exact avoids misrouting maintainers.

Suggested patch 
-# e2e-launchable-smoke job in nightly-e2e.yaml.
+# launchable-smoke-e2e job in nightly-e2e.yaml.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# NOTE: This does NOT test the community install path (launch-plugin.sh).
# For validating what Brev Launchable users actually get, see the
# e2e-launchable-smoke job in nightly-e2e.yaml.
# NOTE: This does NOT test the community install path (launch-plugin.sh).
# For validating what Brev Launchable users actually get, see the
# launchable-smoke-e2e job in nightly-e2e.yaml.
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/e2e-branch-validation.yaml around lines 25 - 27, Update
the header comment that currently references "e2e-launchable-smoke" to the
correct job identifier "launchable-smoke-e2e" so the note matches this PR's
context; search for the string "e2e-launchable-smoke" in the header comment near
the top of .github/workflows/e2e-branch-validation.yaml and replace it with
"launchable-smoke-e2e" to avoid misrouting maintainers.

#
# Test suites:
# full — Install → onboard → sandbox verify → live inference
Expand Down Expand Up @@ -98,11 +117,11 @@ permissions:
pull-requests: write

concurrency:
group: e2e-brev-${{ inputs.pr_number || github.run_id }}
group: e2e-branch-validation-${{ inputs.pr_number || github.run_id }}
cancel-in-progress: true

jobs:
e2e-brev:
e2e-branch-validation:
# if: github.repository == 'NVIDIA/NemoClaw' # Disabled for fork testing — re-enable before merge
runs-on: ubuntu-latest
timeout-minutes: 90
Expand Down Expand Up @@ -182,7 +201,7 @@ jobs:
LAUNCHABLE_SETUP_SCRIPT: ${{ inputs.setup_script_url || '' }}
BREV_PROVIDER: gcp
KEEP_ALIVE: ${{ inputs.keep_alive }}
run: npx vitest run --project e2e-brev --reporter=verbose
run: npx vitest run --project e2e-branch-validation --reporter=verbose

- name: Update check run (completed)
if: always() && inputs.pr_number != '' && env.CHECK_RUN_ID != ''
Expand Down Expand Up @@ -260,6 +279,6 @@ jobs:
if: failure()
uses: actions/upload-artifact@v4
with:
name: e2e-brev-logs
name: e2e-branch-validation-logs
path: /tmp/brev-e2e-*.log
if-no-files-found: ignore
57 changes: 56 additions & 1 deletion .github/workflows/nightly-e2e.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
# credential-migration-e2e Validates legacy ~/.nemoclaw/credentials.json migration to the
# OpenShell gateway, secure zero-fill on unlink, allowlist filter
# on non-credential env keys, and symlink-safe deletion.
# launchable-smoke-e2e Community install path (brev-launchable-ci-cpu.sh) on ubuntu-latest.
# gpu-e2e Local Ollama inference on an NVKS ephemeral GPU runner.
# gpu-double-onboard-e2e Ollama proxy token consistency after re-onboard (#2553).
# notify-on-failure Auto-creates a GitHub issue when any E2E job fails.
Expand Down Expand Up @@ -53,7 +54,7 @@ on:
upgrade-stale-sandbox-e2e, rebuild-hermes-e2e, double-onboard-e2e,
onboard-repair-e2e, onboard-resume-e2e, runtime-overrides-e2e,
credential-sanitization-e2e, telegram-injection-e2e,
overlayfs-autofix-e2e, gpu-e2e, gpu-double-onboard-e2e
overlayfs-autofix-e2e, launchable-smoke-e2e, gpu-e2e, gpu-double-onboard-e2e
required: false
type: string
default: ""
Expand Down Expand Up @@ -1278,6 +1279,59 @@ jobs:
/tmp/nemoclaw-e2e-onboard-negative.log
if-no-files-found: ignore

# ── Launchable Install-Flow Smoke Test ─────────────────────────
# Validates the community install path (brev-launchable-ci-cpu.sh) end-to-end.
# The launchable script has ZERO Brev dependencies — it's a generic Ubuntu
# bootstrap script that runs on ubuntu-latest. Catches regressions like the
# Apr 20-25 Brev outage (#2472, #2482) and container reachability fallback (#2425).
# See: issue #2599
launchable-smoke-e2e:
if: >-
github.repository == 'NVIDIA/NemoClaw' &&
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',launchable-smoke-e2e,'))
runs-on: ubuntu-latest
Comment on lines +1288 to +1294

@coderabbitai coderabbitai Bot Apr 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add launchable-smoke-e2e to report-to-pr.needs to avoid incomplete PR results.

Line 1288 introduces a new runnable job, but report-to-pr (Lines 1531-1555) does not depend on it. That can post a PR summary before this job finishes and omit its status.

🔧 Proposed fix 
   report-to-pr:
     runs-on: ubuntu-latest
     needs:
       [
         cloud-e2e,
         cloud-onboard-e2e,
         cloud-inference-e2e,
         skill-agent-e2e,
         docs-validation-e2e,
         messaging-providers-e2e,
         token-rotation-e2e,
         sandbox-survival-e2e,
         hermes-e2e,
         skip-permissions-e2e,
         sandbox-operations-e2e,
         inference-routing-e2e,
         network-policy-e2e,
         deployment-services-e2e,
         diagnostics-e2e,
         snapshot-commands-e2e,
         shields-config-e2e,
         rebuild-openclaw-e2e,
         upgrade-stale-sandbox-e2e,
         rebuild-hermes-e2e,
         overlayfs-autofix-e2e,
+        launchable-smoke-e2e,
         gpu-e2e,
       ]
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
launchable-smoke-e2e:
if: >-
github.repository == 'NVIDIA/NemoClaw' &&
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',launchable-smoke-e2e,'))
runs-on: ubuntu-latest
report-to-pr:
runs-on: ubuntu-latest
needs:
[
cloud-e2e,
cloud-onboard-e2e,
cloud-inference-e2e,
skill-agent-e2e,
docs-validation-e2e,
messaging-providers-e2e,
token-rotation-e2e,
sandbox-survival-e2e,
hermes-e2e,
skip-permissions-e2e,
sandbox-operations-e2e,
inference-routing-e2e,
network-policy-e2e,
deployment-services-e2e,
diagnostics-e2e,
snapshot-commands-e2e,
shields-config-e2e,
rebuild-openclaw-e2e,
upgrade-stale-sandbox-e2e,
rebuild-hermes-e2e,
overlayfs-autofix-e2e,
launchable-smoke-e2e,
gpu-e2e,
]
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/nightly-e2e.yaml around lines 1288 - 1294, The new job
launchable-smoke-e2e must be added to the dependency list for the report-to-pr
job so the PR summary waits for it; update the report-to-pr job's needs array
(referencing the job name report-to-pr and the new launchable-smoke-e2e job) to
include "launchable-smoke-e2e" alongside the existing needs entries so
report-to-pr runs only after that job completes.

timeout-minutes: 30
steps:
- name: Checkout
uses: actions/checkout@v6

- name: Run launchable install-flow smoke test
env:
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
NEMOCLAW_NON_INTERACTIVE: "1"
NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
NEMOCLAW_SANDBOX_NAME: "e2e-launchable"
NEMOCLAW_RECREATE_SANDBOX: "1"
SKIP_DOCKER_PULL: "1"
GITHUB_TOKEN: ${{ github.token }}
run: bash test/e2e/test-launchable-smoke.sh

- name: Upload install log on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: launchable-smoke-install-log
path: /tmp/nemoclaw-launchable-install.log
if-no-files-found: ignore

- name: Upload onboard log on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: launchable-smoke-onboard-log
path: /tmp/nemoclaw-launchable-onboard.log
if-no-files-found: ignore

- name: Upload test log on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: launchable-smoke-test-log
path: /tmp/nemoclaw-launchable-test.log
if-no-files-found: ignore

Comment on lines +1288 to +1334

@coderabbitai coderabbitai Bot Apr 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

cfg="$(fd -a '^\.coderabbit\.yaml$' | head -n1 || true)"
if [ -z "$cfg" ]; then
  echo "ERROR: .coderabbit.yaml not found"
  exit 1
fi

echo "== Checking .coderabbit.yaml for launchable smoke coverage =="
rg -n -C3 'path_instructions|launchable-smoke-e2e|test/e2e/test-launchable-smoke.sh|brev-launchable-ci-cpu.sh' "$cfg"

echo "== Checking coverage validator test references =="
testfile="$(fd -a '^validate-e2e-coverage\.test\.ts$' | head -n1 || true)"
if [ -n "$testfile" ]; then
  rg -n -C3 'launchable-smoke-e2e|test-launchable-smoke' "$testfile"
else
  echo "WARN: validate-e2e-coverage.test.ts not found"
fi

Repository: NVIDIA/NemoClaw

Length of output: 319

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Find .coderabbit.yaml in root with proper hidden file handling
cfg="$(find . -maxdepth 1 -name '.coderabbit.yaml' | head -n1 || true)"
if [ -z "$cfg" ]; then
  echo "ERROR: .coderabbit.yaml not found in repository root"
  ls -la . | head -20
  exit 1
fi

echo "== .coderabbit.yaml found at: $cfg =="
echo ""
echo "== Searching for launchable-smoke-e2e coverage in .coderabbit.yaml =="
rg 'launchable-smoke-e2e' "$cfg" || echo "WARN: launchable-smoke-e2e not found in .coderabbit.yaml"

echo ""
echo "== Checking full path_instructions section for E2E jobs =="
rg -A 200 'path_instructions:' "$cfg" | head -100

echo ""
echo "== Checking if test script exists =="
test_script="test/e2e/test-launchable-smoke.sh"
if [ -f "$test_script" ]; then
  echo "✓ Test script exists: $test_script"
else
  echo "✗ Test script NOT found: $test_script"
fi

echo ""
echo "== Checking validate-e2e-coverage.test.ts =="
testfile="$(find . -name 'validate-e2e-coverage.test.ts' | head -n1 || true)"
if [ -n "$testfile" ]; then
  echo "Found: $testfile"
  rg 'launchable-smoke' "$testfile" || echo "WARN: launchable-smoke reference not found in test file"
else
  echo "WARN: validate-e2e-coverage.test.ts not found"
fi

Repository: NVIDIA/NemoClaw

Length of output: 5756

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== Full validate-e2e-coverage.test.ts content =="
cat test/validate-e2e-coverage.test.ts

echo ""
echo "== Search for other E2E job entries in .coderabbit.yaml =="
rg -n 'e2e|smoke|test/' .coderabbit.yaml | head -50

echo ""
echo "== Check for notify-on-failure in .coderabbit.yaml =="
rg -B 5 -A 20 'notify-on-failure' .coderabbit.yaml || echo "WARN: notify-on-failure not found"

Repository: NVIDIA/NemoClaw

Length of output: 13179

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== Checking notify-on-failure in nightly-e2e.yaml =="
rg -B 3 -A 30 'notify-on-failure:' .github/workflows/nightly-e2e.yaml | head -80

Repository: NVIDIA/NemoClaw

Length of output: 1063

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== Check if launchable-smoke-e2e appears in nightly-e2e.yaml =="
rg 'launchable-smoke-e2e' .github/workflows/nightly-e2e.yaml

echo ""
echo "== Full notify-on-failure needs list =="
rg -A 50 'notify-on-failure:' .github/workflows/nightly-e2e.yaml | grep -A 40 'needs:'

Repository: NVIDIA/NemoClaw

Length of output: 1654

Add path_instructions entry to .coderabbit.yaml for launchable-smoke-e2e.

The new job was correctly added to notify-on-failure needs list, but no corresponding path_instructions mapping exists in .coderabbit.yaml. Per coding guidelines, every new E2E job requires a path mapping for the source files it covers.

Add an entry that maps test/e2e/test-launchable-smoke.sh to the job name launchable-smoke-e2e in the path_instructions block, following the pattern used by other E2E jobs. Note: the cross-validation test in validate-e2e-coverage.test.ts only warns about uncovered jobs (does not fail), so this mapping must be added manually.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/nightly-e2e.yaml around lines 1288 - 1334, Add a new
path_instructions mapping in .coderabbit.yaml associating the test file
test/e2e/test-launchable-smoke.sh with the job name launchable-smoke-e2e; locate
the existing path_instructions block and add an entry following the same pattern
as other E2E jobs (map the file path to the job identifier
"launchable-smoke-e2e") so the notify-on-failure coverage includes this new job.

# ── GPU E2E (Ollama local inference) ──────────────────────────
# Runs on an NVKS ephemeral GPU runner (RTX Pro 6000, 36 GB VRAM).
# Each job gets a fresh VM — no state leakage between runs.
Expand Down Expand Up @@ -1423,6 +1477,7 @@ jobs:
credential-sanitization-e2e,
telegram-injection-e2e,
overlayfs-autofix-e2e,
launchable-smoke-e2e,
gpu-e2e,
gpu-double-onboard-e2e,
]
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Tests are organized into three Vitest projects defined in `vitest.config.ts`:

1. **`cli`** — `test/**/*.test.{js,ts}` — integration tests for CLI behavior
2. **`plugin`** — `nemoclaw/src/**/*.test.ts` — unit tests co-located with source
3. **`e2e-brev`** — `test/e2e/brev-e2e.test.js` — cloud E2E (requires `BREV_API_TOKEN`)
3. **`e2e-branch-validation`** — `test/e2e/brev-e2e.test.ts` — validates a branch from source on ephemeral Brev instance (requires `BREV_API_TOKEN`)

When writing tests:

Expand Down
15 changes: 11 additions & 4 deletions test/e2e/brev-e2e.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,20 @@
// SPDX-License-Identifier: Apache-2.0

/**
* Ephemeral Brev E2E test suite.
* Branch Validation E2E — installs NemoClaw FROM SOURCE on a fresh Brev instance.
*
* Creates a fresh Brev instance via the launchable bootstrap path, bootstraps it,
* runs E2E tests remotely, then tears it down.
* Answers: "Does this branch work if you install from source on a clean machine?"
*
* Creates a fresh Brev instance, rsyncs the checked-out branch code, runs
* install.sh from source, onboards a sandbox, then executes the selected test
* suite against the live environment. Tears down the instance when done.
*
* NOTE: This does NOT test the community Launchable install path
* (launch-plugin.sh). For that, see test-launchable-smoke.sh wired into
* nightly-e2e.yaml.
*
* Intended to be run from CI via:
* npx vitest run --project e2e-brev
* npx vitest run --project e2e-branch-validation
*
* Required env vars:
* NVIDIA_API_KEY — passed to VM for inference config during onboarding
Expand Down
Loading
Loading