Skip to content

test(e2e): add model router routed guard for #3255 #3594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jyaunches wants to merge 3 commits into from
Closed

test(e2e): add model router routed guard for #3255 #3594

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b016b7c
test(e2e): add model router routed guard
jyaunches May 15, 2026
4be46fa
fix: address routed inference feedback
jyaunches May 15, 2026
af5b343
fix(e2e): satisfy model router guard checks
jyaunches May 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion .coderabbit.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,8 @@ reviews:
- `cloud-e2e` — full onboard + cloud inference
- `sandbox-operations-e2e` — multi-sandbox lifecycle
- `rebuild-openclaw-e2e` — workspace state survives rebuild
- `channels-stop-start-e2e` — channel disable/enable lifecycle across
rebuild with cached Telegram credentials
- `messaging-compatible-endpoint-e2e` — Telegram + compatible endpoint
inference.local regression
- `hermes-discord-e2e` — Hermes Discord config schema + placeholder
Expand All @@ -193,9 +195,37 @@ reviews:

To run selectively:
```
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=cloud-e2e,sandbox-operations-e2e,rebuild-openclaw-e2e,messaging-compatible-endpoint-e2e,hermes-discord-e2e,hermes-slack-e2e,openshell-gateway-upgrade-e2e
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=cloud-e2e,sandbox-operations-e2e,rebuild-openclaw-e2e,channels-stop-start-e2e,messaging-compatible-endpoint-e2e,hermes-discord-e2e,hermes-slack-e2e,openshell-gateway-upgrade-e2e
```

- path: "src/lib/onboard/channel-state.ts"
instructions: &e2e-channel-stop-start |
This file controls disabled channel resolution used during onboard
and rebuild. Changes affect whether `channels stop` persists across
sandbox destroy/recreate and whether `channels start` reattaches
cached credentials.

**E2E test recommendation:**
- `channels-stop-start-e2e` — Telegram channel stop/start lifecycle
across rebuild with cached credentials

To run selectively:
```
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=channels-stop-start-e2e
```

- path: "src/lib/state/onboard-session.ts"
instructions: *e2e-channel-stop-start

- path: "src/lib/actions/sandbox/rebuild.ts"
instructions: *e2e-channel-stop-start

- path: "src/lib/commands/sandbox/channels/**"
instructions: *e2e-channel-stop-start

- path: "test/e2e/test-channels-stop-start.sh"
instructions: *e2e-channel-stop-start

- path: "src/lib/actions/inference-set.ts"
instructions: |
This file switches the OpenShell inference route and patches the
Expand Down
58 changes: 58 additions & 0 deletions .github/workflows/brev-nightly-e2e.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

name: brev-nightly-e2e

# Runs Brev launchable validation as a standalone nightly workflow while the
# main nightly-e2e workflow remains isolated from Brev platform flakiness.
#
# Suites:
# all credential-sanitization + telegram-injection
# messaging-providers Telegram + Discord provider/L7 proxy validation
# full install/onboard/inference/CLI path

on:
schedule:
- cron: "0 6 * * *"
workflow_dispatch:
inputs:
branch:
description: "Branch to test (default: ref used for this dispatch; schedule always tests main)"
required: false
default: ""
launchable_id:
description: "Published launchable ID override (empty = workflow/test default)"
required: false
default: ""
keep_alive:
description: "Keep Brev instances alive after tests (for SSH debugging)"
required: false
type: boolean
default: false

permissions:
contents: read

concurrency:
group: brev-nightly-e2e-${{ github.event_name }}-${{ github.event_name == 'workflow_dispatch' && github.ref || 'schedule' }}
cancel-in-progress: true

jobs:
brev-nightly-e2e:
if: github.repository == 'NVIDIA/NemoClaw'
strategy:
fail-fast: false
matrix:
test_suite: [all, messaging-providers, full]
uses: ./.github/workflows/e2e-branch-validation.yaml
with:
branch: ${{ github.event_name == 'schedule' && 'main' || inputs.branch || github.ref_name }}
test_suite: ${{ matrix.test_suite }}
use_launchable: true
use_published_launchable: true
launchable_id: ${{ github.event_name == 'workflow_dispatch' && inputs.launchable_id || '' }}
Comment on lines +52 to +53

@coderabbitai coderabbitai Bot May 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "== Declared workflow_call inputs (.github/workflows/e2e-branch-validation.yaml) =="
awk '
  /^[[:space:]]*workflow_call:[[:space:]]*$/ {in_wc=1; next}
  in_wc && /^[[:space:]]*inputs:[[:space:]]*$/ {in_inputs=1; next}
  in_inputs {
    if ($0 ~ /^[[:space:]]{4}[a-zA-Z0-9_-]+:[[:space:]]*$/) {
      key=$1; sub(":", "", key); print key
    } else if ($0 ~ /^[[:space:]]{2}[a-zA-Z0-9_-]+:[[:space:]]*$/) {
      exit
    }
  }
' .github/workflows/e2e-branch-validation.yaml | sort -u

echo
echo "== Inputs passed by brev-nightly-e2e caller =="
awk '
  /^[[:space:]]*with:[[:space:]]*$/ {in_with=1; next}
  in_with {
    if ($0 ~ /^[[:space:]]{6}[a-zA-Z0-9_-]+:[[:space:]]*/) {
      key=$1; sub(":", "", key); print key
    } else if ($0 ~ /^[[:space:]]{4}(secrets|permissions|strategy|if|uses|name):[[:space:]]*$/) {
      exit
    }
  }
' .github/workflows/brev-nightly-e2e.yaml | sort -u

Repository: NVIDIA/NemoClaw

Length of output: 281

Remove all undeclared reusable-workflow inputs.

The following inputs passed at lines 49-54 are not declared by .github/workflows/e2e-branch-validation.yaml: branch, keep_alive, launchable_id, test_suite, use_launchable, and use_published_launchable. Only secrets is declared as a workflow_call input. Passing undeclared inputs will cause workflow-call validation failure and prevent this job from executing.

🧰 Tools
🪛 actionlint (1.7.12)

[error] 52-52: input "use_published_launchable" is not defined in "./.github/workflows/e2e-branch-validation.yaml" reusable workflow. defined inputs are "branch", "brev_create_timeout_seconds", "brev_gpu_min_vram", "brev_gpu_name", "brev_gpu_type", "brev_provider", "keep_alive", "pr_number", "setup_script_url", "test_suite", "use_launchable"

(workflow-call)

[error] 53-53: input "launchable_id" is not defined in "./.github/workflows/e2e-branch-validation.yaml" reusable workflow. defined inputs are "branch", "brev_create_timeout_seconds", "brev_gpu_min_vram", "brev_gpu_name", "brev_gpu_type", "brev_provider", "keep_alive", "pr_number", "setup_script_url", "test_suite", "use_launchable"

(workflow-call)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/brev-nightly-e2e.yaml around lines 52 - 53, The job is
passing undeclared reusable-workflow inputs (branch, keep_alive, launchable_id,
test_suite, use_launchable, use_published_launchable) to the called workflow;
either remove these parameters from the call in
.github/workflows/brev-nightly-e2e.yaml or add matching workflow_call input
declarations in .github/workflows/e2e-branch-validation.yaml. Locate the call
site where these keys are supplied (the entries for branch, keep_alive,
launchable_id, test_suite, use_launchable, use_published_launchable) and delete
them if they aren’t needed, or open e2e-branch-validation.yaml and add
corresponding inputs under workflow_call with appropriate defaults/types so the
call validates.

keep_alive: ${{ github.event_name == 'workflow_dispatch' && inputs.keep_alive || false }}
secrets:
BREV_API_KEY: ${{ secrets.BREV_API_KEY }}
BREV_ORG_ID: ${{ secrets.BREV_ORG_ID }}
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
154 changes: 137 additions & 17 deletions .github/workflows/e2e-scenarios.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# SPDX-License-Identifier: Apache-2.0
#
# Scenario-based E2E. Runs a single setup scenario by id against the
# matching runner; can also validate resolution / coverage via --plan-only.
# matching runner and uploads runtime artifacts for debugging.
#
# Manual-only (workflow_dispatch) while scenario-based coverage migrates.
# Existing nightly-e2e / macos-e2e / wsl-e2e workflows remain unchanged.
Expand All @@ -16,14 +16,6 @@ on:
description: "Scenario id (e.g. ubuntu-repo-cloud-openclaw)"
required: true
type: string
plan_only:
description: "Resolve and print plan only (no install/onboard/suites)"
required: false
default: "false"
type: choice
options:
- "true"
- "false"
suite_filter:
description: "Comma-separated suite ids to run (optional; defaults to the scenario's full suite list)"
required: false
Expand Down Expand Up @@ -55,9 +47,9 @@ jobs:
SCENARIO: ${{ github.event.inputs.scenario }}
run: |
case "${SCENARIO}" in
macos-*) echo "runner=macos-latest" >> "$GITHUB_OUTPUT" ;;
macos-*) echo "runner=macos-26" >> "$GITHUB_OUTPUT" ;;
wsl-*) echo "runner=windows-latest" >> "$GITHUB_OUTPUT" ;;
gpu-*) echo "runner=self-hosted" >> "$GITHUB_OUTPUT" ;;
gpu-*) echo "runner=linux-amd64-gpu-rtxpro6000-latest-1" >> "$GITHUB_OUTPUT" ;;
ubuntu-*|brev-*) echo "runner=ubuntu-latest" >> "$GITHUB_OUTPUT" ;;
*)
echo "::error::Unknown scenario prefix for runner selection: ${SCENARIO}" >&2
Expand All @@ -68,38 +60,165 @@ jobs:
run-scenario:
needs: resolve-runner
runs-on: ${{ needs.resolve-runner.outputs.runner }}
timeout-minutes: 45
timeout-minutes: 90
env:
WSL_DISTRO: Ubuntu
NEMOCLAW_RECREATE_SANDBOX: "1"
steps:
- name: Force LF line endings for WSL checkout
if: startsWith(github.event.inputs.scenario, 'wsl-')
shell: powershell
run: git config --global core.autocrlf false

- uses: actions/checkout@v4

- name: Set up Node
if: ${{ !startsWith(github.event.inputs.scenario, 'wsl-') }}
uses: actions/setup-node@v6
with:
node-version: 22
cache: npm

- name: Install root dependencies
if: ${{ !startsWith(github.event.inputs.scenario, 'wsl-') }}
run: npm ci --ignore-scripts

- name: Render coverage report
if: ${{ !startsWith(github.event.inputs.scenario, 'wsl-') }}
run: |
mkdir -p .e2e
bash test/e2e/runtime/coverage-report.sh > .e2e/coverage.md
echo '## E2E scenario coverage' >> "$GITHUB_STEP_SUMMARY"
cat .e2e/coverage.md >> "$GITHUB_STEP_SUMMARY"

- name: Show resolved plan
run: |
bash test/e2e/runtime/run-scenario.sh "${{ github.event.inputs.scenario }}" --plan-only

- name: Run scenario
if: github.event.inputs.plan_only != 'true'
if: ${{ !startsWith(github.event.inputs.scenario, 'wsl-') }}
env:
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
E2E_SUITE_FILTER: ${{ github.event.inputs.suite_filter }}
run: |
bash test/e2e/runtime/run-scenario.sh "${{ github.event.inputs.scenario }}"

- name: Resolve workspace paths for WSL
if: startsWith(github.event.inputs.scenario, 'wsl-')
shell: powershell
run: |
$winPath = "${{ github.workspace }}"
$drive = $winPath.Substring(0,1).ToLower()
$rest = $winPath.Substring(2).Replace('\','/')
$wslCheckoutPath = "/mnt/$drive$rest"
$wslWorkdir = "/tmp/nemoclaw-scenario-wsl/${env:GITHUB_RUN_ID}-${env:GITHUB_RUN_ATTEMPT}"
"WSL_CHECKOUT_DIR=$wslCheckoutPath" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
"WSL_WORKDIR=$wslWorkdir" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append

- name: Ensure Ubuntu WSL exists
if: startsWith(github.event.inputs.scenario, 'wsl-')
shell: powershell
run: |
wsl --list --verbose 2>&1 | Out-Default
$null = wsl -d $env:WSL_DISTRO -- echo ok 2>&1
if ($LASTEXITCODE -ne 0) {
wsl --install -d $env:WSL_DISTRO --no-launch --web-download
wsl -d $env:WSL_DISTRO -- bash -c 'echo distro initialised'
}
wsl --set-default $env:WSL_DISTRO

- name: Install WSL dependencies
if: startsWith(github.event.inputs.scenario, 'wsl-')
shell: powershell
run: |
$script = @'
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive
printf '%s\n' 'Acquire::ForceIPv4 "true";' 'Acquire::Retries "5";' >/etc/apt/apt.conf.d/99github-actions-network
apt-get update
apt-get install -y bash ca-certificates curl git jq lsb-release make python3 python3-pip rsync tar unzip xz-utils
if ! docker info >/dev/null 2>&1; then
apt-get install -y docker.io
service docker start || /etc/init.d/docker start || true
timeout 30 bash -c 'until docker info >/dev/null 2>&1; do sleep 2; done'
fi
curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
apt-get install -y nodejs
node --version
npm --version
docker --version
docker info >/dev/null
'@
$tmp = "$env:RUNNER_TEMP\wsl-step.sh"
[IO.File]::WriteAllText($tmp, ($script -replace "`r",""), (New-Object System.Text.UTF8Encoding $false))
$wslTmp = wsl -d $env:WSL_DISTRO -- wslpath -u ($tmp -replace '\\','/')
wsl -d $env:WSL_DISTRO -- bash -l $wslTmp

- name: Copy checkout into WSL ext4 workspace
if: startsWith(github.event.inputs.scenario, 'wsl-')
shell: powershell
run: |
$script = @"
set -euo pipefail
rm -rf '$env:WSL_WORKDIR'
mkdir -p /tmp/nemoclaw-scenario-wsl
rsync -a --no-owner --no-group --delete --exclude '/node_modules/' --exclude '/nemoclaw/node_modules/' --exclude '/nemoclaw-blueprint/.venv/' '$env:WSL_CHECKOUT_DIR'/ '$env:WSL_WORKDIR'/
git config --global --add safe.directory '$env:WSL_WORKDIR'
git -C '$env:WSL_WORKDIR' reset --hard HEAD
git -C '$env:WSL_WORKDIR' clean -ffdx
"@
$tmp = "$env:RUNNER_TEMP\wsl-step.sh"
[IO.File]::WriteAllText($tmp, ($script -replace "`r",""), (New-Object System.Text.UTF8Encoding $false))
$wslTmp = wsl -d $env:WSL_DISTRO -- wslpath -u ($tmp -replace '\\','/')
wsl -d $env:WSL_DISTRO -- bash -l $wslTmp

- name: Install root dependencies in WSL
if: startsWith(github.event.inputs.scenario, 'wsl-')
shell: powershell
run: |
$script = @"
set -euo pipefail
cd '$env:WSL_WORKDIR'
npm ci --ignore-scripts
mkdir -p .e2e
bash test/e2e/runtime/coverage-report.sh > .e2e/coverage.md
"@
$tmp = "$env:RUNNER_TEMP\wsl-step.sh"
[IO.File]::WriteAllText($tmp, ($script -replace "`r",""), (New-Object System.Text.UTF8Encoding $false))
$wslTmp = wsl -d $env:WSL_DISTRO -- wslpath -u ($tmp -replace '\\','/')
wsl -d $env:WSL_DISTRO -- bash -l $wslTmp

- name: Run scenario in WSL
if: startsWith(github.event.inputs.scenario, 'wsl-')
shell: powershell
env:
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
E2E_SUITE_FILTER: ${{ github.event.inputs.suite_filter }}
run: |
$script = @"
set -euo pipefail
cd '$env:WSL_WORKDIR'
export NVIDIA_API_KEY='$env:NVIDIA_API_KEY'
export E2E_SUITE_FILTER='$env:E2E_SUITE_FILTER'
export NEMOCLAW_RECREATE_SANDBOX='$env:NEMOCLAW_RECREATE_SANDBOX'
bash test/e2e/runtime/run-scenario.sh '${{ github.event.inputs.scenario }}'
"@
$tmp = "$env:RUNNER_TEMP\wsl-step.sh"
[IO.File]::WriteAllText($tmp, ($script -replace "`r",""), (New-Object System.Text.UTF8Encoding $false))
$wslTmp = wsl -d $env:WSL_DISTRO -- wslpath -u ($tmp -replace '\\','/')
wsl -d $env:WSL_DISTRO -- bash -l $wslTmp

- name: Copy WSL artifacts back to checkout
if: always() && startsWith(github.event.inputs.scenario, 'wsl-')
shell: powershell
run: |
$script = @"
set -euo pipefail
mkdir -p '$env:WSL_CHECKOUT_DIR/.e2e' '$env:WSL_CHECKOUT_DIR/test/e2e/logs'
if [ -d '$env:WSL_WORKDIR/.e2e' ]; then rsync -a '$env:WSL_WORKDIR/.e2e'/ '$env:WSL_CHECKOUT_DIR/.e2e'/; fi
if [ -d '$env:WSL_WORKDIR/test/e2e/logs' ]; then rsync -a '$env:WSL_WORKDIR/test/e2e/logs'/ '$env:WSL_CHECKOUT_DIR/test/e2e/logs'/; fi
"@
$tmp = "$env:RUNNER_TEMP\wsl-step.sh"
[IO.File]::WriteAllText($tmp, ($script -replace "`r",""), (New-Object System.Text.UTF8Encoding $false))
$wslTmp = wsl -d $env:WSL_DISTRO -- wslpath -u ($tmp -replace '\\','/')
wsl -d $env:WSL_DISTRO -- bash -l $wslTmp

- name: Upload scenario artifacts
if: always()
uses: actions/upload-artifact@v4
Expand All @@ -110,3 +229,4 @@ jobs:
test/e2e/logs/
if-no-files-found: warn
retention-days: 14
include-hidden-files: true
44 changes: 43 additions & 1 deletion .github/workflows/nightly-e2e.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ on:
credential-sanitization-e2e, telegram-injection-e2e,
overlayfs-autofix-e2e, device-auth-health-e2e,
launchable-smoke-e2e, gpu-e2e, gpu-double-onboard-e2e,
brave-search-e2e
channels-stop-start-e2e, brave-search-e2e
required: false
type: string
default: ""
Expand Down Expand Up @@ -371,6 +371,45 @@ jobs:
path: /tmp/nemoclaw-e2e-messaging-compatible-endpoint-install.log
if-no-files-found: ignore

# ── Channels stop/start lifecycle E2E (#3462 Test 1) ─────────
# Regression coverage for #3453 (channels stop must actually disable the
# channel across rebuild) and #3381 (channels start must re-attach from
# the cached credential). Telegram-only — Discord/Slack walk the same
# disabledChannels filter; telegram is the cheapest regression gate.
channels-stop-start-e2e:
if: >-
github.repository == 'NVIDIA/NemoClaw' &&
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',channels-stop-start-e2e,'))
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout
uses: actions/checkout@v6
with:
ref: ${{ inputs.target_ref || github.ref }}

- name: Run channels stop/start lifecycle E2E test
env:
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
NEMOCLAW_NON_INTERACTIVE: "1"
NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
NEMOCLAW_POLICY_TIER: "open"
NEMOCLAW_SANDBOX_NAME: "e2e-channels-stop-start"
GITHUB_TOKEN: ${{ github.token }}
TELEGRAM_BOT_TOKEN: "test-fake-telegram-token-stop-start-e2e"
TELEGRAM_ALLOWED_IDS: "123456789"
run: bash test/e2e/test-channels-stop-start.sh

- name: Upload install log on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: install-log-channels-stop-start
path: /tmp/nemoclaw-e2e-install.log
if-no-files-found: ignore

# ── Brave Search E2E (#2687) ─────────────────────────────────
# Validates the full Brave Search path with a real BRAVE_API_KEY:
# non-interactive onboard auto-enables web search, the brave network
Expand Down Expand Up @@ -1976,6 +2015,7 @@ jobs:
docs-validation-e2e,
messaging-providers-e2e,
messaging-compatible-endpoint-e2e,
channels-stop-start-e2e,
brave-search-e2e,
kimi-inference-compat-e2e,
token-rotation-e2e,
Expand Down Expand Up @@ -2068,6 +2108,7 @@ jobs:
docs-validation-e2e,
messaging-providers-e2e,
messaging-compatible-endpoint-e2e,
channels-stop-start-e2e,
brave-search-e2e,
kimi-inference-compat-e2e,
token-rotation-e2e,
Expand Down Expand Up @@ -2217,6 +2258,7 @@ jobs:
docs-validation-e2e,
messaging-providers-e2e,
messaging-compatible-endpoint-e2e,
channels-stop-start-e2e,
brave-search-e2e,
kimi-inference-compat-e2e,
token-rotation-e2e,
Expand Down
Loading
Loading