Skip to content

Revert "ci(nightly): enable brev-e2e job with long-lived BREV_API_TOKEN (#3350)" #3372

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ericksoa wants to merge 1 commit into from
Closed

Revert "ci(nightly): enable brev-e2e job with long-lived BREV_API_TOKEN (#3350)" #3372

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
160 changes: 35 additions & 125 deletions .github/workflows/e2e-branch-validation.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,26 +3,18 @@

name: e2e-branch-validation

# ─── Branch Validation E2E ───────────────────────────────────────────
# ─── Branch Validation E2E ───────────────────────────────────────────────────
#
# PURPOSE: Validates a specific git branch by rsync-ing it onto a real Brev
# cloud instance provisioned from the PUBLISHED NemoClaw launchable image,
# then running the selected security / messaging / full suites against it.
# Answers: "Does this branch work on the image customers actually get?"
# PURPOSE: Validates a specific git branch by installing NemoClaw FROM SOURCE
# on an ephemeral Brev cloud instance. Answers the question: "Does this branch
# work if you install from source on a clean machine?"
#
# HOW IT WORKS:
# 1. Provisions a Brev CPU instance from the published NemoClaw launchable
# (env-3Azt0aYgVNFEuz7opyx3gscmowS) — a pre-baked GCP image built 3x
# daily by brevdev/nemoclaw-image with NemoClaw, OpenShell, and the
# sandbox image already on disk. Boots in ~2 min.
# 2. Rsyncs the checked-out branch code over the image's ~/NemoClaw.
# 3. Rebuilds CLI dist/, runs onboard, and executes the selected suite.
# 4. Tears down the instance (unless keep_alive=true).
#
# FALLBACK: set use_published_launchable=false to provision a bare Ubuntu
# VM and bootstrap via scripts/brev-launchable-ci-cpu.sh. Slower (~5 min)
# and does NOT test the image itself — useful only for validating changes
# to the startup script without touching the published image.
# 1. Provisions a fresh Brev CPU instance (~4 vCPU, 16 GB RAM)
# 2. Rsyncs the checked-out branch code to the VM
# 3. Runs install.sh from source + onboards a sandbox
# 4. Executes the selected test suite against the live sandbox
# 5. Tears down the instance (unless keep_alive=true)
#
# WHEN TO USE:
# - Before merging a PR that touches onboard, sandbox, security, or infra
Expand Down Expand Up @@ -53,27 +45,14 @@ name: e2e-branch-validation
# all — Runs credential-sanitization + telegram-injection (NOT full,
# which destroys the sandbox the security tests need).
#
# Required secrets: BREV_API_KEY, BREV_ORG_ID, NVIDIA_API_KEY
# BREV_API_KEY — long-lived Brev API key (format: bak-...). Replaces the
# legacy short-lived BREV_API_TOKEN refresh-token secret.
# BREV_ORG_ID — Brev organization id (format: org-...). Required alongside
# the API key for brev login.
#
# Optional inputs:
# launchable_id — override the published launchable. Default (empty →
# env-3Azt0aYgVNFEuz7opyx3gscmowS) matches docs/deployment/
# brev-web-ui.md. Override to test a staging image.
# Required secrets: BREV_API_TOKEN, NVIDIA_API_KEY
# Instance cost: Brev CPU credits (~$0.10/run for 4x16 instance)

on:
workflow_dispatch:
inputs:
branch:
description: "Branch to test (default: the branch this dispatch targets via --ref)"
required: false
default: ""
pr_number:
description: "PR number (resolves branch automatically, overrides branch input)"
description: "PR number (resolves branch automatically)"
required: false
default: ""
test_suite:
Expand All @@ -88,24 +67,19 @@ on:
- messaging-providers
- all
use_launchable:
description: "Deprecated — always true. Kept for backward-compat with existing dispatches."
description: "Use CI launchable (true) or bare brev create + brev-setup.sh (false)"
required: false
type: boolean
default: true
use_published_launchable:
description: "Provision from the published NemoClaw launchable image (recommended) vs. the repo-local startup-script fallback"
required: false
type: boolean
default: true
launchable_id:
description: "Published launchable ID (default: the public NemoClaw launchable env-3Azt0aYgVNFEuz7opyx3gscmowS)"
required: false
default: ""
keep_alive:
description: "Keep Brev instance alive after tests (for SSH debugging)"
required: false
type: boolean
default: false
brev_token:
description: "Brev refresh token (overrides BREV_API_TOKEN secret if provided)"
required: false
default: ""
workflow_call:
inputs:
branch:
Expand All @@ -123,14 +97,6 @@ on:
required: false
type: boolean
default: true
use_published_launchable:
required: false
type: boolean
default: true
launchable_id:
required: false
type: string
default: ""
setup_script_url:
required: false
type: string
Expand All @@ -140,9 +106,7 @@ on:
type: boolean
default: true
secrets:
BREV_API_KEY:
required: true
BREV_ORG_ID:
BREV_API_TOKEN:
required: true
NVIDIA_API_KEY:
required: true
Expand All @@ -158,14 +122,7 @@ concurrency:

jobs:
e2e-branch-validation:
# Allow both upstream (NVIDIA/NemoClaw) and the known CI fork (jyaunches/NemoClaw).
# The fork is used to validate the long-lived BREV_API_KEY + nightly wiring
# before the secret is rotated into the upstream repo. Other forks get a no-op
# skip so community PRs that touch this file don't try to run against someone
# else's Brev account.
if: >-
github.repository == 'NVIDIA/NemoClaw' ||
github.repository == 'jyaunches/NemoClaw'
# if: github.repository == 'NVIDIA/NemoClaw' # Disabled for fork testing — re-enable before merge
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
Expand All @@ -181,15 +138,7 @@ jobs:
- name: Checkout target branch
uses: actions/checkout@v6
with:
# Precedence:
# 1. PR-resolved branch (from pr_number lookup above)
# 2. Explicit branch input (workflow_dispatch / workflow_call)
# 3. github.ref_name (the ref that was dispatched against) —
# this is what `gh workflow run --ref X` passes. Without this
# fallback, workflow_dispatch without explicit inputs silently
# checks out 'main' even when the dispatch targeted a feature
# branch, making branch changes invisible to the run.
ref: ${{ env.RESOLVED_BRANCH || inputs.branch || github.ref_name }}
ref: ${{ env.RESOLVED_BRANCH || inputs.branch || 'main' }}

- name: Create check run (pending)
if: inputs.pr_number != ''
Expand All @@ -215,79 +164,40 @@ jobs:

- name: Install Brev CLI
env:
# GitHub Actions inputs are plaintext run metadata; Brev credentials
# must only come from repository/org secrets so they remain masked.
BREV_API_KEY: ${{ secrets.BREV_API_KEY }}
BREV_ORG_ID: ${{ secrets.BREV_ORG_ID }}
BREV_API_TOKEN: ${{ inputs.brev_token || secrets.BREV_API_TOKEN }}
run: |
# Brev CLI v0.6.324+ — first release to support `brev login --api-key`
# for long-lived CI auth (bak-... keys). Earlier v0.6.322 pin required
# writing ~/.brev/credentials.json by hand because `brev login` had
# been removed in #1470 and only supported interactive OAuth.
curl -fsSL -o /tmp/brev.tar.gz "https://github.com/brevdev/brev-cli/releases/download/v0.6.324/brev-cli_0.6.324_linux_amd64.tar.gz"
# Brev CLI v0.6.322+ — CPU instances use `brev search cpu | brev create`
# Startup scripts use `brev create --startup-script @file` (not brev start --cpu)
curl -fsSL -o /tmp/brev.tar.gz "https://github.com/brevdev/brev-cli/releases/download/v0.6.322/brev-cli_0.6.322_linux_amd64.tar.gz"
tar -xzf /tmp/brev.tar.gz -C /usr/local/bin brev
chmod +x /usr/local/bin/brev
if [ -z "${BREV_API_KEY:-}" ]; then
echo "::error::BREV_API_KEY is empty — cannot authenticate Brev CLI."
# Brev CLI does not read BREV_API_TOKEN from env — it requires
# ~/.brev/credentials.json. The login call was removed in #1470
# (374a847d), breaking CI. Write the credentials file so `brev ls`
# works in the test harness's hasAuthenticatedBrev check.
if [ -z "${BREV_API_TOKEN:-}" ]; then
echo "::error::BREV_API_TOKEN is empty — cannot authenticate Brev CLI."
exit 1
fi
if [ -z "${BREV_ORG_ID:-}" ]; then
echo "::error::BREV_ORG_ID is empty — required for API-key login."
exit 1
fi
# Long-lived API-key login. Persists api_key + api_key_org_id into
# ~/.brev/credentials.json; no browser, no refresh-token expiry.
brev login --api-key "$BREV_API_KEY" --org-id "$BREV_ORG_ID"
mkdir -p ~/.brev
umask 077
printf '{"refresh_token":"%s"}' "$BREV_API_TOKEN" > ~/.brev/credentials.json
chmod 600 ~/.brev/credentials.json
# Skip the first-run onboarding wizard that blocks on stdin (removed in #1470)
# Skip the first-run onboarding wizard that blocks on stdin (also removed in #1470)
printf '{"step":1,"hasRunBrevShell":true,"hasRunBrevOpen":true}' > ~/.brev/onboarding_step.json
# Verify credentials with brev ls. The Brev backend periodically
# returns 'context deadline exceeded' on the workspaces/list
# endpoint under platform load — retry a few times before
# surfacing as a hard failure. Observed on 2026-05-11 during PR
# #3350 validation: up to ~30 min windows where brev ls failed
# across multiple orgs with working credentials.
for attempt in 1 2 3 4 5; do
if brev ls >/dev/null 2>/tmp/brev-ls-err; then
echo "brev ls succeeded on attempt $attempt"
break
fi
err=$(cat /tmp/brev-ls-err)
if [ "$attempt" -eq 5 ]; then
echo "::warning::brev ls failed 5 times (last error: $err) — proceeding anyway; the test harness will retry with brev refresh if needed."
break
fi
echo "brev ls attempt $attempt failed ($err) — retrying in 15s..."
sleep 15
done
brev ls >/dev/null

- name: Install dependencies
run: npm install --ignore-scripts

- name: Build CLI (dist/)
# The Brev E2E harness invokes bin/nemoclaw.js locally for pre-flight
# sandbox-name validation before touching Brev. bin/nemoclaw.js
# requires '../dist/nemoclaw', which is produced by the `prepare`
# npm script — skipped above because --ignore-scripts is set to
# avoid the git-hook installer running in CI. Build explicitly.
run: npm run build:cli

- name: Run ephemeral Brev E2E
env:
BREV_API_KEY: ${{ secrets.BREV_API_KEY }}
BREV_ORG_ID: ${{ secrets.BREV_ORG_ID }}
BREV_API_TOKEN: ${{ inputs.brev_token || secrets.BREV_API_TOKEN }}
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
GITHUB_TOKEN: ${{ github.token }}
INSTANCE_NAME: e2e-pr-${{ inputs.pr_number || github.run_id }}
TEST_SUITE: ${{ inputs.test_suite }}
USE_LAUNCHABLE: ${{ inputs.use_launchable && '1' || '0' }}
# Use the published NemoClaw launchable (pre-baked GCP image, ~2 min
# boot) by default. Set to '0' to fall back to the repo-local
# startup-script path (bare VM, ~5 min bootstrap).
USE_PUBLISHED_LAUNCHABLE: ${{ inputs.use_published_launchable && '1' || '0' }}
# Launchable ID override. Empty string → test harness default
# (env-3Azt0aYgVNFEuz7opyx3gscmowS, the public NemoClaw launchable).
BREV_LAUNCHABLE_ID: ${{ inputs.launchable_id || '' }}
LAUNCHABLE_SETUP_SCRIPT: ${{ inputs.setup_script_url || '' }}
BREV_PROVIDER: gcp
KEEP_ALIVE: ${{ inputs.keep_alive }}
Expand Down
56 changes: 1 addition & 55 deletions .github/workflows/nightly-e2e.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,6 @@
# OpenShell gateway, secure zero-fill on unlink, allowlist filter
# on non-credential env keys, and symlink-safe deletion.
# launchable-smoke-e2e Community install path (brev-launchable-ci-cpu.sh) on ubuntu-latest.
# brev-e2e Ephemeral Brev cloud instance end-to-end — provisions a real
# Brev VM from the PUBLISHED NemoClaw launchable image
# (env-3Azt0aYgVNFEuz7opyx3gscmowS, rebuilt 3x daily by
# brevdev/nemoclaw-image), rsyncs main over it, runs the
# selected test suites, and tears down. This is the only
# coverage that validates the image customers actually get.
# Requires BREV_API_KEY (long-lived bak-... CI key) + BREV_ORG_ID.
# gpu-e2e Local Ollama inference on an NVKS ephemeral GPU runner.
# gpu-double-onboard-e2e Ollama proxy token consistency after re-onboard (#2553).
# notify-on-failure Auto-creates a GitHub issue when any E2E job fails.
Expand Down Expand Up @@ -92,7 +85,7 @@ on:
onboard-repair-e2e, onboard-resume-e2e, runtime-overrides-e2e,
credential-sanitization-e2e, telegram-injection-e2e,
overlayfs-autofix-e2e, device-auth-health-e2e,
launchable-smoke-e2e, brev-e2e, gpu-e2e, gpu-double-onboard-e2e,
launchable-smoke-e2e, gpu-e2e, gpu-double-onboard-e2e,
brave-search-e2e
required: false
type: string
Expand Down Expand Up @@ -1780,50 +1773,6 @@ jobs:
path: /tmp/nemoclaw-launchable-test.log
if-no-files-found: ignore

# ── Ephemeral Brev E2E (real cloud instance + launchable) ────────
# Provisions a fresh Brev CPU instance via the CI launchable, installs
# NemoClaw from main, and runs the selected security + messaging suites.
# Calls the reusable e2e-branch-validation.yaml workflow.
#
# Matrix rationale:
# - `all` covers credential-sanitization + telegram-injection
# (the two security regression suites that need a shared sandbox).
# - `messaging-providers` creates its own sandbox and exercises the
# Telegram + Discord L7 proxy chain.
# - `full` is the end-to-end install → onboard → inference → CLI path.
# Listed last because it destroys its sandbox on completion.
#
# Cost: ~$0.10 per Brev instance × 3 suites ≈ $9/month at nightly cadence.
# Secrets: BREV_API_KEY (long-lived bak-... CI key), BREV_ORG_ID (org-...),
# NVIDIA_API_KEY.
brev-e2e:
if: >-
github.repository == 'NVIDIA/NemoClaw' &&
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',brev-e2e,'))
strategy:
fail-fast: false
matrix:
test_suite: [all, messaging-providers, full]
uses: ./.github/workflows/e2e-branch-validation.yaml
with:
# Respect the selected ref for manual dispatches; scheduled runs execute
# on main, so this remains main there.
branch: ${{ github.ref_name }}
test_suite: ${{ matrix.test_suite }}
use_launchable: true
# Provision from the published NemoClaw launchable image (pre-baked
# GCP image, ~2 min boot), not the startup-script fallback. Empty
# launchable_id → harness default env-3Azt0aYgVNFEuz7opyx3gscmowS.
use_published_launchable: true
launchable_id: ""
keep_alive: false
secrets:
BREV_API_KEY: ${{ secrets.BREV_API_KEY }}
BREV_ORG_ID: ${{ secrets.BREV_ORG_ID }}
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}

# ── GPU E2E (Ollama local inference) ──────────────────────────
# Runs on an NVKS ephemeral GPU runner (RTX Pro 6000, 36 GB VRAM).
# Each job gets a fresh VM — no state leakage between runs.
Expand Down Expand Up @@ -1981,7 +1930,6 @@ jobs:
overlayfs-autofix-e2e,
device-auth-health-e2e,
launchable-smoke-e2e,
brev-e2e,
gpu-e2e,
gpu-double-onboard-e2e,
]
Expand Down Expand Up @@ -2074,7 +2022,6 @@ jobs:
overlayfs-autofix-e2e,
device-auth-health-e2e,
launchable-smoke-e2e,
brev-e2e,
gpu-e2e,
gpu-double-onboard-e2e,
]
Expand Down Expand Up @@ -2215,7 +2162,6 @@ jobs:
overlayfs-autofix-e2e,
device-auth-health-e2e,
launchable-smoke-e2e,
brev-e2e,
gpu-e2e,
gpu-double-onboard-e2e,
]
Expand Down
Loading
Loading