Repository navigation
ci(e2e): add exact-SHA Brev Launchable E2E lane #7043
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 9 commits
Commits
Show all changes
40 commits
Select commit
Hold shift + click to select a range
d8ed815
ci(e2e): add exact image qualification handoff
cjagwani 752e561
test(e2e): extract qualification controller fixtures
cjagwani 47cfafd
fix(e2e): preserve corrupt qualification state
cjagwani 9a74662
fix(e2e): validate qualification state provenance
cjagwani 5ebf2a0
ci(e2e): add launchable evidence contract
cjagwani 325daa7
chore(e2e): defer launchable receipt validator
cjagwani 9f60a94
Merge remote-tracking branch 'origin/main' into codex/launchable-imag…
jyaunches 00b1b78
ci(e2e): qualify exact staging Brev Launchables
jyaunches 8cd0ad5
ci(e2e): reuse image dispatch credential
jyaunches be0aa7a
fix(e2e): preserve selected candidate provenance
jyaunches de8ab46
fix(e2e): harden staging launchable qualification
jyaunches 7b54051
fix(e2e): harden staging launchable qualification
jyaunches c9b9468
test(e2e): map exact launchable fast coverage
jyaunches 1c5a1cf
merge: refresh PR #7043 from main
cv 08a83aa
fix(e2e): retry initial Brev cleanup inventory
cv f80d1e6
Merge branch 'main' into codex/launchable-image-manifest-handoff
cv 234af91
Merge branch 'main' into codex/launchable-image-manifest-handoff
cv cb10195
Merge branch 'main' into codex/launchable-image-manifest-handoff
cv 4f96e55
Merge branch 'main' into codex/launchable-image-manifest-handoff
cjagwani 5e96bdf
fix(e2e): trust main for launchable qualification
cjagwani 572b9f6
Merge remote-tracking branch 'origin/main' into codex/pr7043-trust-fix
cjagwani 639c6f8
fix(e2e): follow private file module rename
cjagwani 5bb8d82
Merge remote-tracking branch 'origin/codex/launchable-image-manifest-…
cjagwani cc60309
fix(e2e): source launchable secrets from environment
cjagwani c24b186
Merge remote-tracking branch 'origin/main' into codex/pr7043-trust-fix
cjagwani 8b13312
ci(e2e): gate staging qualification activation
cjagwani fdb4e9d
merge: refresh PR #7043 with main
cjagwani a8f0d89
merge: refresh PR #7043 with main
cjagwani 5c90776
merge: refresh PR #7043 with main
cjagwani ace0573
test(e2e): reuse full suite for Brev Launchable
jyaunches c589eef
test(e2e): satisfy test conditional guardrail
jyaunches 68d004f
fix(e2e): secure Launchable qualification evidence
cjagwani eea94c7
Merge remote-tracking branch 'origin/main' into codex/pr-7043-simplify
jyaunches e61f90f
fix(e2e): align Brev workspace states
cjagwani e88cc4a
ci(e2e): simplify Brev Launchable coverage
jyaunches 47409d5
refactor(e2e): simplify exact-image Launchable lane
cjagwani 31648ae
Merge branch 'codex/launchable-image-manifest-handoff' of github.com:…
jyaunches 89747ee
Merge remote-tracking branch 'origin/codex/launchable-image-manifest-…
jyaunches 646a3f8
fix(e2e): avoid producer contents lookup
jyaunches 4b95fa4
fix(e2e): verify launchable image directly
jyaunches File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,309 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| name: E2E / Exact Staging Brev Launchable | ||
| run-name: Exact staging Launchable qualification for ${{ inputs.candidate_sha }} | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| candidate_sha: | ||
| description: Exact lowercase 40-character NemoClaw commit SHA to qualify. | ||
| required: true | ||
| type: string | ||
| reason: | ||
| description: Audit reason for starting this exact-image Launchable qualification. | ||
| required: true | ||
| type: string | ||
| workflow_call: | ||
| inputs: | ||
| candidate_sha: | ||
| description: Exact lowercase 40-character NemoClaw commit SHA to qualify. | ||
| required: true | ||
| type: string | ||
| reason: | ||
| description: Audit reason for starting this exact-image Launchable qualification. | ||
| required: true | ||
| type: string | ||
| secrets: | ||
| NEMOCLAW_IMAGE_DISPATCH_TOKEN: | ||
| required: true | ||
| BREV_API_KEY: | ||
| required: true | ||
| BREV_ORG_ID: | ||
| required: true | ||
| NVIDIA_INFERENCE_API_KEY: | ||
| required: true | ||
|
|
||
| permissions: {} | ||
|
|
||
| concurrency: | ||
| group: brev-launchable-qualification-${{ inputs.candidate_sha }} | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| preflight: | ||
| name: Validate exact qualification request | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 10 | ||
| permissions: | ||
| contents: read | ||
| steps: | ||
| - name: Checkout trusted controller | ||
| uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 | ||
| with: | ||
| ref: ${{ github.workflow_sha }} | ||
| persist-credentials: false | ||
|
|
||
| - name: Setup Node | ||
| uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.0.0 | ||
| with: | ||
| node-version: "22" | ||
|
|
||
| - name: Validate candidate and maintainer authority | ||
| env: | ||
| ACTOR: ${{ github.triggering_actor }} | ||
| CANDIDATE_SHA: ${{ inputs.candidate_sha }} | ||
| EVENT_NAME: ${{ github.event_name }} | ||
| GITHUB_TOKEN: ${{ github.token }} | ||
| QUALIFICATION_REASON: ${{ inputs.reason }} | ||
| REQUESTER_RUN_ATTEMPT: ${{ github.run_attempt }} | ||
| REQUESTER_RUN_ID: ${{ github.run_id }} | ||
| REQUEST_REF: ${{ github.ref }} | ||
| WORKFLOW_SHA: ${{ github.workflow_sha }} | ||
| run: >- | ||
| node --experimental-strip-types --no-warnings | ||
| tools/e2e/exact-image-qualification-controller.mts | ||
| --mode preflight | ||
| --actor "$ACTOR" | ||
| --candidate-sha "$CANDIDATE_SHA" | ||
| --event-name "$EVENT_NAME" | ||
| --reason "$QUALIFICATION_REASON" | ||
| --ref "$REQUEST_REF" | ||
| --requester-run-attempt "$REQUESTER_RUN_ATTEMPT" | ||
| --requester-run-id "$REQUESTER_RUN_ID" | ||
| --workflow-sha "$WORKFLOW_SHA" | ||
|
|
||
| qualify: | ||
| name: Build, deploy, and test exact staging image | ||
| needs: preflight | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 120 | ||
| environment: | ||
| name: approve-brev-launchable-qualification | ||
| deployment: false | ||
| permissions: | ||
| contents: read | ||
| steps: | ||
| - name: Checkout trusted controller | ||
| uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 | ||
| with: | ||
| ref: ${{ github.workflow_sha }} | ||
| persist-credentials: false | ||
|
|
||
| - name: Setup Node | ||
| uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.0.0 | ||
| with: | ||
| node-version: "22" | ||
|
|
||
| - id: workspace | ||
| name: Create private evidence workspace | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| work_dir="$(mktemp -d "${RUNNER_TEMP}/nemoclaw-image-qualification.XXXXXX")" | ||
| chmod 700 "$work_dir" | ||
| printf 'work_dir=%s\n' "$work_dir" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - id: start | ||
| name: Dispatch exact qualification image build | ||
| env: | ||
| ACTOR: ${{ github.triggering_actor }} | ||
| CANDIDATE_SHA: ${{ inputs.candidate_sha }} | ||
| EVENT_NAME: ${{ github.event_name }} | ||
| GITHUB_TOKEN: ${{ github.token }} | ||
| NEMOCLAW_IMAGE_QUALIFICATION_TOKEN: ${{ secrets.NEMOCLAW_IMAGE_DISPATCH_TOKEN }} | ||
| QUALIFICATION_REASON: ${{ inputs.reason }} | ||
| REQUESTER_RUN_ATTEMPT: ${{ github.run_attempt }} | ||
| REQUESTER_RUN_ID: ${{ github.run_id }} | ||
| REQUEST_REF: ${{ github.ref }} | ||
| WORKFLOW_SHA: ${{ github.workflow_sha }} | ||
| run: >- | ||
| node --experimental-strip-types --no-warnings | ||
| tools/e2e/exact-image-qualification-controller.mts | ||
| --mode start | ||
| --actor "$ACTOR" | ||
| --candidate-sha "$CANDIDATE_SHA" | ||
| --event-name "$EVENT_NAME" | ||
| --reason "$QUALIFICATION_REASON" | ||
| --ref "$REQUEST_REF" | ||
| --requester-run-attempt "$REQUESTER_RUN_ATTEMPT" | ||
| --requester-run-id "$REQUESTER_RUN_ID" | ||
| --workflow-sha "$WORKFLOW_SHA" | ||
| --work-dir "${{ steps.workspace.outputs.work_dir }}" | ||
|
|
||
| - name: Wait for the bound producer run | ||
| env: | ||
| NEMOCLAW_IMAGE_QUALIFICATION_TOKEN: ${{ secrets.NEMOCLAW_IMAGE_DISPATCH_TOKEN }} | ||
| run: >- | ||
| node --experimental-strip-types --no-warnings | ||
| tools/e2e/exact-image-qualification-controller.mts | ||
| --mode wait | ||
| --work-dir "${{ steps.workspace.outputs.work_dir }}" | ||
|
|
||
| - name: Download and verify the bound manifest artifact | ||
| env: | ||
| NEMOCLAW_IMAGE_QUALIFICATION_TOKEN: ${{ secrets.NEMOCLAW_IMAGE_DISPATCH_TOKEN }} | ||
| run: >- | ||
| node --experimental-strip-types --no-warnings | ||
| tools/e2e/exact-image-qualification-controller.mts | ||
| --mode download | ||
| --work-dir "${{ steps.workspace.outputs.work_dir }}" | ||
|
|
||
| - name: Validate the exact image manifest | ||
| env: | ||
| CANDIDATE_SHA: ${{ inputs.candidate_sha }} | ||
| CORRELATION_ID: ${{ steps.start.outputs.correlation_id }} | ||
| IMAGE_REPOSITORY_SHA: ${{ steps.start.outputs.image_repository_sha }} | ||
| PRODUCER_RUN_ATTEMPT: ${{ steps.start.outputs.producer_run_attempt }} | ||
| PRODUCER_RUN_ID: ${{ steps.start.outputs.producer_run_id }} | ||
| REQUESTER_RUN_ATTEMPT: ${{ github.run_attempt }} | ||
| REQUESTER_RUN_ID: ${{ github.run_id }} | ||
| run: >- | ||
| node --experimental-strip-types --no-warnings | ||
| tools/e2e/validate-exact-image-manifest.mts | ||
| --manifest "${{ steps.workspace.outputs.work_dir }}/nemoclaw-image-manifest.v1.json" | ||
| --output "${{ steps.workspace.outputs.work_dir }}/validated-manifest.v1.json" | ||
| --nemoclaw-sha "$CANDIDATE_SHA" | ||
| --requester-run-id "$REQUESTER_RUN_ID" | ||
| --requester-run-attempt "$REQUESTER_RUN_ATTEMPT" | ||
| --correlation-id "$CORRELATION_ID" | ||
| --image-repository-sha "$IMAGE_REPOSITORY_SHA" | ||
| --producer-run-id "$PRODUCER_RUN_ID" | ||
| --producer-run-attempt "$PRODUCER_RUN_ATTEMPT" | ||
|
|
||
| - name: Record accepted provenance and hashes | ||
| run: >- | ||
| node --experimental-strip-types --no-warnings | ||
| tools/e2e/exact-image-qualification-controller.mts | ||
| --mode finalize | ||
| --work-dir "${{ steps.workspace.outputs.work_dir }}" | ||
|
|
||
| - name: Install pinned Brev CLI | ||
| env: | ||
| BREV_CLI_SHA256: 5a6e70374db9be33f85f299161733b4a8409840d47638c781429b96e8d53704f | ||
| BREV_CLI_VERSION: 0.6.330 | ||
| run: | | ||
| set -euo pipefail | ||
| archive="${RUNNER_TEMP}/brev-cli.tar.gz" | ||
| curl -fsSL -o "$archive" "https://github.com/brevdev/brev-cli/releases/download/v${BREV_CLI_VERSION}/brev-cli_${BREV_CLI_VERSION}_linux_amd64.tar.gz" | ||
| printf '%s %s\n' "$BREV_CLI_SHA256" "$archive" | sha256sum -c - | ||
| tar -xzf "$archive" -C "${RUNNER_TEMP}" brev | ||
| mkdir -p "${RUNNER_TEMP}/bin" | ||
| install -m 0755 "${RUNNER_TEMP}/brev" "${RUNNER_TEMP}/bin/brev" | ||
| printf '%s\n' "${RUNNER_TEMP}/bin" >> "$GITHUB_PATH" | ||
|
|
||
| - name: Authenticate Brev CLI | ||
| env: | ||
| BREV_API_KEY: ${{ secrets.BREV_API_KEY }} | ||
| BREV_ORG_ID: ${{ secrets.BREV_ORG_ID }} | ||
| run: | | ||
| set -euo pipefail | ||
| brev login --api-key "$BREV_API_KEY" --org-id "$BREV_ORG_ID" | ||
| brev ls --json | jq -e '.workspaces | type == "array"' >/dev/null | ||
|
|
||
| - id: brev-workspace | ||
| name: Bind unique staging workspace name | ||
| env: | ||
| CANDIDATE_SHA: ${{ inputs.candidate_sha }} | ||
| run: | | ||
| set -euo pipefail | ||
| short_sha="${CANDIDATE_SHA:0:8}" | ||
| instance_name="nclaw-e2e-${short_sha}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" | ||
| printf 'instance_name=%s\n' "$instance_name" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Deploy the configured staging Launchable | ||
| env: | ||
| BREV_LAUNCHABLE_ID: ${{ vars.NEMOCLAW_STAGING_LAUNCHABLE_ID }} | ||
| INSTANCE_NAME: ${{ steps.brev-workspace.outputs.instance_name }} | ||
| WORK_DIR: ${{ steps.workspace.outputs.work_dir }} | ||
| run: tools/e2e/brev-launchable-runtime.sh deploy | ||
|
|
||
| - name: Verify exact boot image and run preinstalled-image smoke | ||
| env: | ||
| CANDIDATE_SHA: ${{ inputs.candidate_sha }} | ||
| INSTANCE_NAME: ${{ steps.brev-workspace.outputs.instance_name }} | ||
| NVIDIA_INFERENCE_API_KEY: ${{ secrets.NVIDIA_INFERENCE_API_KEY }} | ||
| VALIDATED_MANIFEST: ${{ steps.workspace.outputs.work_dir }}/validated-manifest.v1.json | ||
| WORK_DIR: ${{ steps.workspace.outputs.work_dir }} | ||
| run: tools/e2e/brev-launchable-runtime.sh qualify | ||
|
|
||
| - name: Redact runtime evidence | ||
| if: ${{ always() && steps.workspace.outputs.work_dir != '' }} | ||
| env: | ||
| NVIDIA_INFERENCE_API_KEY: ${{ secrets.NVIDIA_INFERENCE_API_KEY }} | ||
| WORK_DIR: ${{ steps.workspace.outputs.work_dir }} | ||
| run: | | ||
| set -euo pipefail | ||
| python3 - <<'PY' | ||
| import os | ||
| from pathlib import Path | ||
|
|
||
| secret = os.environ.get("NVIDIA_INFERENCE_API_KEY", "") | ||
| root = Path(os.environ["WORK_DIR"]) | ||
| if secret: | ||
| for name in ("brev-prerequisites.log", "brev-quickstart.log", "brev-inference-pong.json", "brev-agent-response.log"): | ||
| path = root / name | ||
| if path.is_file(): | ||
| path.write_text(path.read_text(encoding="utf-8", errors="replace").replace(secret, "[REDACTED]"), encoding="utf-8") | ||
| PY | ||
|
|
||
| - name: Delete staging workspace and verify absence | ||
| if: ${{ always() && steps.workspace.outputs.work_dir != '' && steps.brev-workspace.outputs.instance_name != '' }} | ||
| env: | ||
| INSTANCE_NAME: ${{ steps.brev-workspace.outputs.instance_name }} | ||
| WORK_DIR: ${{ steps.workspace.outputs.work_dir }} | ||
| run: tools/e2e/brev-launchable-runtime.sh cleanup | ||
|
|
||
| - name: Cancel an incomplete producer run | ||
| if: ${{ always() && steps.workspace.outputs.work_dir != '' && steps.start.outcome != 'skipped' }} | ||
| continue-on-error: true | ||
| env: | ||
| NEMOCLAW_IMAGE_QUALIFICATION_TOKEN: ${{ secrets.NEMOCLAW_IMAGE_DISPATCH_TOKEN }} | ||
| run: >- | ||
| node --experimental-strip-types --no-warnings | ||
| tools/e2e/exact-image-qualification-controller.mts | ||
| --mode cancel | ||
| --work-dir "${{ steps.workspace.outputs.work_dir }}" | ||
|
|
||
| - name: Upload exact staging Launchable evidence | ||
| if: ${{ always() && steps.workspace.outputs.work_dir != '' }} | ||
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | ||
| with: | ||
| name: brev-launchable-qualification-${{ inputs.candidate_sha }}-${{ github.run_id }} | ||
| path: | | ||
| ${{ steps.workspace.outputs.work_dir }}/dispatch-intent.v1.json | ||
| ${{ steps.workspace.outputs.work_dir }}/dispatch-reconciliation.v1.json | ||
| ${{ steps.workspace.outputs.work_dir }}/controller-state.json | ||
| ${{ steps.workspace.outputs.work_dir }}/controller-state.corrupt-*.json | ||
| ${{ steps.workspace.outputs.work_dir }}/nemoclaw-image-handoff.zip | ||
| ${{ steps.workspace.outputs.work_dir }}/nemoclaw-image-manifest.v1.json | ||
| ${{ steps.workspace.outputs.work_dir }}/validated-manifest.v1.json | ||
| ${{ steps.workspace.outputs.work_dir }}/qualification-evidence.v1.json | ||
| ${{ steps.workspace.outputs.work_dir }}/brev-deploy-request.json | ||
| ${{ steps.workspace.outputs.work_dir }}/brev-workspace-ready.json | ||
| ${{ steps.workspace.outputs.work_dir }}/brev-provision.json | ||
| ${{ steps.workspace.outputs.work_dir }}/brev-boot-image.json | ||
| ${{ steps.workspace.outputs.work_dir }}/brev-identity-evidence.json | ||
| ${{ steps.workspace.outputs.work_dir }}/brev-prerequisites.log | ||
| ${{ steps.workspace.outputs.work_dir }}/brev-quickstart.log | ||
| ${{ steps.workspace.outputs.work_dir }}/brev-inference-pong.json | ||
| ${{ steps.workspace.outputs.work_dir }}/brev-agent-response.log | ||
| ${{ steps.workspace.outputs.work_dir }}/brev-smoke-evidence.json | ||
| ${{ steps.workspace.outputs.work_dir }}/brev-cleanup-evidence.json | ||
| if-no-files-found: warn | ||
| retention-days: 90 | ||
|
|
||
| - name: Remove private evidence workspace | ||
| if: ${{ always() && steps.workspace.outputs.work_dir != '' }} | ||
| run: rm -rf -- "${{ steps.workspace.outputs.work_dir }}" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.