Skip to content

fix(health): add verifyDeployment() and fix false 'Health Offline' on 401 #3128

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jyaunches merged 19 commits into from
May 6, 2026
Merged

fix(health): add verifyDeployment() and fix false 'Health Offline' on 401 #3128

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
05db612
fix(health): add verifyDeployment() and fix false 'Health Offline' on…
jyaunches Apr 29, 2026
658a0c5
ci: add cross-repo image rebuild trigger and pin launchable to stable
jyaunches Apr 29, 2026
7b9f547
test(e2e): add device-auth-health regression test for #2342
jyaunches May 6, 2026
e0c18a2
fix(e2e): increase device-auth-health timeout to 30m
jyaunches May 6, 2026
2f47f5e
fix(e2e): use SSH for sandbox exec and increase timeout
jyaunches May 6, 2026
b07cfaf
ci(nightly): migrate E2E jobs to NVIDIA self-hosted runners
jyaunches May 6, 2026
6e6f24d
fix(e2e): fix PATH and install flow in device-auth-health test
jyaunches May 6, 2026
0e17cce
Merge remote-tracking branch 'origin/main' into issue-2342-brev-launc…
jyaunches May 6, 2026
16fa34c
fix(e2e): make sandbox exec probes non-fatal when SSH unavailable
jyaunches May 6, 2026
5ec077e
style(e2e): apply shfmt formatting to device-auth-health test
jyaunches May 6, 2026
7340c70
fix: use named sandbox exec format for health probes
jyaunches May 6, 2026
6ac5434
style: apply shfmt formatting to brev-launchable-ci-cpu.sh
jyaunches May 6, 2026
9228360
style: fix shfmt case-indent (-ci) in brev-launchable-ci-cpu.sh
jyaunches May 6, 2026
dc8ce62
test: update cli.test.ts probe mocks for new curl health pattern
jyaunches May 6, 2026
d39a353
fix(health): treat gateway version as cosmetic, not a health signal
jyaunches May 6, 2026
2a1aa17
refactor: remove out-of-scope CI changes from health fix PR
jyaunches May 6, 2026
674831e
Merge branch 'main' into issue-2342-brev-launchable-version-pin-and-g…
cv May 6, 2026
3075a70
Merge branch 'main' into issue-2342-brev-launchable-version-pin-and-g…
prekshivyas May 6, 2026
24c3ef0
fix: address CodeRabbit review feedback
jyaunches May 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 72 additions & 33 deletions .github/workflows/nightly-e2e.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,8 @@ on:
rebuild-hermes-stale-base-e2e, double-onboard-e2e,
onboard-repair-e2e, onboard-resume-e2e, runtime-overrides-e2e,
credential-sanitization-e2e, telegram-injection-e2e,
overlayfs-autofix-e2e, launchable-smoke-e2e, gpu-e2e, gpu-double-onboard-e2e
overlayfs-autofix-e2e, device-auth-health-e2e,
launchable-smoke-e2e, gpu-e2e, gpu-double-onboard-e2e
required: false
type: string
default: ""
Expand All @@ -80,7 +81,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',cloud-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 45
steps:
- name: Checkout
Expand Down Expand Up @@ -114,7 +115,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',cloud-onboard-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 45
steps:
- name: Checkout
Expand Down Expand Up @@ -150,7 +151,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',cloud-inference-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 30
steps:
- name: Checkout
Expand Down Expand Up @@ -182,7 +183,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',skill-agent-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 30
steps:
- name: Checkout
Expand Down Expand Up @@ -214,7 +215,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',docs-validation-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 15
steps:
- name: Checkout
Expand Down Expand Up @@ -249,7 +250,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',messaging-providers-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 45
steps:
- name: Checkout
Expand Down Expand Up @@ -287,7 +288,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',messaging-compatible-endpoint-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 45
steps:
- name: Checkout
Expand Down Expand Up @@ -323,7 +324,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',token-rotation-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 45
steps:
- name: Checkout
Expand Down Expand Up @@ -361,7 +362,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',sandbox-survival-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 30
steps:
- name: Checkout
Expand Down Expand Up @@ -394,7 +395,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',issue-2478-crash-loop-recovery-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 30
steps:
- name: Checkout
Expand Down Expand Up @@ -427,7 +428,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',hermes-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 60
steps:
- name: Checkout
Expand Down Expand Up @@ -462,7 +463,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',hermes-discord-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 60
steps:
- name: Checkout
Expand Down Expand Up @@ -502,7 +503,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',sandbox-operations-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 60
steps:
- name: Checkout
Expand Down Expand Up @@ -748,7 +749,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',inference-routing-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 30
steps:
- name: Checkout
Expand Down Expand Up @@ -779,7 +780,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',network-policy-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 45
steps:
- name: Checkout
Expand Down Expand Up @@ -812,7 +813,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',deployment-services-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 60
steps:
- name: Checkout
Expand Down Expand Up @@ -844,7 +845,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',diagnostics-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 45
steps:
- name: Checkout
Expand Down Expand Up @@ -878,7 +879,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',credential-migration-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 30
steps:
- name: Checkout
Expand Down Expand Up @@ -911,7 +912,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',snapshot-commands-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 30
steps:
- name: Checkout
Expand Down Expand Up @@ -943,7 +944,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',shields-config-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 30
steps:
- name: Checkout
Expand Down Expand Up @@ -975,7 +976,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',rebuild-openclaw-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 60
steps:
- name: Checkout
Expand Down Expand Up @@ -1008,7 +1009,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',upgrade-stale-sandbox-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 60
steps:
- name: Checkout
Expand Down Expand Up @@ -1041,7 +1042,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',rebuild-hermes-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 60
steps:
- name: Checkout
Expand Down Expand Up @@ -1074,7 +1075,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',rebuild-hermes-stale-base-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 60
steps:
- name: Checkout
Expand Down Expand Up @@ -1106,7 +1107,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',double-onboard-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 90
steps:
- name: Checkout
Expand Down Expand Up @@ -1143,7 +1144,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',onboard-repair-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 60
steps:
- name: Checkout
Expand Down Expand Up @@ -1180,7 +1181,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',onboard-resume-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 60
steps:
- name: Checkout
Expand Down Expand Up @@ -1217,7 +1218,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',runtime-overrides-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 45
steps:
- name: Checkout
Expand Down Expand Up @@ -1255,7 +1256,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',credential-sanitization-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 60
steps:
- name: Checkout
Expand Down Expand Up @@ -1296,7 +1297,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',telegram-injection-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 60
steps:
- name: Checkout
Expand Down Expand Up @@ -1340,7 +1341,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',overlayfs-autofix-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 45
steps:
- name: Checkout
Expand All @@ -1366,6 +1367,42 @@ jobs:
/tmp/nemoclaw-e2e-onboard-negative.log
if-no-files-found: ignore

# ── Device Auth Health Probe (#2342) ────────────────────────────
# Regression test for #2342: verifies health probes work correctly when
# device auth is enabled (the default). Previously `curl -sf` treated
# HTTP 401 as failure, causing false "Health Offline" readings.
# Validates: /health returns 200, / returns 401, status != Offline,
# gateway recovery with device auth, port forward liveness.
device-auth-health-e2e:
if: >-
github.repository == 'NVIDIA/NemoClaw' &&
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',device-auth-health-e2e,'))
runs-on: linux-amd64-cpu4
timeout-minutes: 30
steps:
- name: Checkout
uses: actions/checkout@v6

- name: Run device auth health E2E
env:
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
NEMOCLAW_NON_INTERACTIVE: "1"
NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
NEMOCLAW_SANDBOX_NAME: "e2e-health-auth"
NEMOCLAW_RECREATE_SANDBOX: "1"
GITHUB_TOKEN: ${{ github.token }}
run: bash test/e2e/test-device-auth-health.sh

- name: Upload install log on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: device-auth-health-install-log
path: /tmp/nemoclaw-e2e-health-install.log
if-no-files-found: ignore
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

# ── Launchable Install-Flow Smoke Test ─────────────────────────
# Validates the community install path (brev-launchable-ci-cpu.sh) end-to-end.
# The launchable script has ZERO Brev dependencies — it's a generic Ubuntu
Expand All @@ -1378,7 +1415,7 @@ jobs:
(github.event_name != 'workflow_dispatch' ||
inputs.jobs == '' ||
contains(format(',{0},', inputs.jobs), ',launchable-smoke-e2e,'))
runs-on: ubuntu-latest
runs-on: linux-amd64-cpu4
timeout-minutes: 30
steps:
- name: Checkout
Expand Down Expand Up @@ -1566,6 +1603,7 @@ jobs:
credential-sanitization-e2e,
telegram-injection-e2e,
overlayfs-autofix-e2e,
device-auth-health-e2e,
launchable-smoke-e2e,
gpu-e2e,
gpu-double-onboard-e2e,
Expand Down Expand Up @@ -1642,6 +1680,7 @@ jobs:
rebuild-hermes-e2e,
rebuild-hermes-stale-base-e2e,
overlayfs-autofix-e2e,
device-auth-health-e2e,
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
gpu-e2e,
]
if: ${{ always() && github.event_name == 'workflow_dispatch' }}
Expand Down
Loading
Loading