ci(e2e): centralize larger runner routing - #7245
Closed
cjagwani wants to merge 1 commit into
Closed
Conversation
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Collaborator
Author
|
/nvskills-ci |
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit ed2ea17 in the TypeScript / code-coverage/cliThe overall coverage in commit ed2ea17 in the Show a code coverage summary of the most impacted files.
Updated |
Contributor
PR Review Advisor — InformationalAdvisor assessment: Informational / high confidence Model lanes
Nemotron output stays in workflow artifacts and does not change the assessment above. E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge. |
wscurran
added
area: ci
23 tasks
21 tasks
cv
added a commit
that referenced
this pull request
Jul 23, 2026
<!-- markdownlint-disable MD041 --> ## Summary <!-- 1-3 plain sentences: what changes and why. Describe before-and-after behavior when it applies. Follow the NemoClaw Writing Guide: https://github.com/NVIDIA/NemoClaw/blob/main/WRITING.md. Do not add unrelated prose cleanup. --> Rebuilds #7245 on current `main` and centralizes larger-runner selection for resource-heavy E2E lanes. Trusted final-main executions can use one administrator-configured label, while unset, PR-checkout, and untrusted contexts continue to use `ubuntu-latest`. Exact-head PR gates now add a bounded swap fallback for the hosted Hermes image-building lanes that intentionally remain on standard runners. ## Related Issue <!-- Fixes #NNN or Closes #NNN. Remove this section if none. --> Part of #7145. Supersedes #7245, whose protected head branch is stacked on obsolete #7244 history. ## Changes <!-- List concrete changes. If this adds an abstraction, configuration, fallback, migration, or compatibility path, name its current requirement and consumer, explain why a direct change is insufficient, and identify the test that protects it. --> - Build one validated runner map in `generate-matrix` before target code is checked out, consuming `E2E_LARGER_RUNNER_LABEL` only for `NVIDIA/NemoClaw` on `refs/heads/main` with no alternate checkout SHA. - Route common egress, Hermes rebuilds, heavy MCP shards, and the Hermes lanes repeatedly interrupted in run 29897237525 through that map. - Keep the OpenClaw matrix counterparts, MCP OpenClaw/dev lanes, and unrelated jobs on standard runners. - For exact-head PR gates only, reuse at least 32 GiB of active swap or provision one fixed 32 GiB file before the hosted Hermes image-building lanes. The setup uses absolute non-interactive executables in a clean environment and fails closed before Vitest. - Exclude scheduled/manual `main`, larger-runner, workflow-managed rebuild, dedicated-runner, MCP dev, and non-Hermes executions from the swap fallback. - Document the runner sizing, concurrency, comparison, fallback, and rollback requirements. - Protect the centralized rollout, exact-head fallback, and trust boundary with executable tests. A workflow-only swap step cannot help an alternate checkout until that workflow revision reaches trusted `main`. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check one tests line and one docs line. Check other lines when applicable. Add every requested justification or approval reference. --> - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent focused review passed with no blockers after exact-head-only activation, absolute privileged executables, a clean fixed environment, and active-swap-safe cleanup were enforced at `af1cb1c77`. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed implementation - Result: `docs-updated` - Evidence: `test/e2e/README.md` documents the larger-runner trust boundary and the bounded exact-head swap fallback, selected Hermes lanes, fail-closed behavior, exclusions, sizing, measurement, and rollback requirements. - Agent: Codex Desktop - PR: #7391 <!-- docs-review-head-sha: 0d92d12 --> <!-- docs-review-agents-blob-sha: 560ff38 --> ## DGX Station Hardware Evidence <!-- Required only when scripts/prepare-dgx-station-host.sh changes. Maintainers must review the linked evidence before approving or merging. This is human-reviewed evidence, not authenticated hardware provenance. Exceptional bypasses use existing repository governance and must be documented on the PR. --> - [ ] Tested on DGX Station - Tested commit: - Station profile/scenario: - Result: - Supporting evidence: ## Verification <!-- Check each applicable item only when supported by the requested evidence. Run targeted tests once per relevant change set and rerun after later edits or hook autofixes that can affect the tested behavior. Do not rerun hook-covered checks. --> - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 59 focused E2E-support tests passed after the current `main` sync. - [x] Applicable broad gate passed — `npm run checks`, `npm run typecheck:cli`, and `npm run docs` passed; Fern reported 0 errors and 2 non-fatal warnings. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * E2E workflows now generate a trusted runner-routing map and use it to route selected “heavy” lanes to a configurable larger Ubuntu runner, with validation and safe fallback to `ubuntu-latest`. * Live Vitest E2E runs can now provision a bounded Hermes swap file beforehand when required, and fail closed if provisioning fails. * **Documentation** * Added guidance for enabling, validating boundaries, and rolling back larger-runner routing. * **Tests** * Added e2e boundary coverage for runner routing and expanded Hermes swap provisioning/selection coverage. * **CI / Chores** * Updated workflow boundary checks and routing assertions to match the new trusted routing behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Charan Jagwani <cjagwani@nvidia.com> Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Signed-off-by: Carlos Villela <cvillela@nvidia.com> Co-authored-by: Charan Jagwani <cjagwani@nvidia.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Centralizes the heavy-lane runner map required by #7145 while leaving the experiment inactive until an administrator configures a GitHub-hosted larger runner. Unset or untrusted configurations keep every eligible lane on
ubuntu-latest, and clearing the configuration variable provides the rollback.Related Issue
Part of #7145
Changes
generate-matrixbefore checking out test code, consumingE2E_LARGER_RUNNER_LABELonly forNVIDIA/NemoClawatrefs/heads/main.common-egress-agent, both Hermes rebuild jobs, and the Hermes/Deep Agents MCP shards through that map; keep MCP OpenClaw/dev and unrelated jobs on their existing assignments.The centralized map and fallback are required by #7145 so the external runner can be activated and rolled back without editing every consumer. Directly changing each
runs-onvalue would make rollout depend on an unavailable label and would require another broad workflow edit to roll back.larger-runner-routing-workflow-boundary.test.tsprotects the current consumer and trust contracts.Type of Change
Quality Gates
Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailablenpx vitest run --project e2e-support test/e2e/support/larger-runner-routing-workflow-boundary.test.ts test/e2e/support/e2e-workflow.test.ts test/e2e/support/mcp-workflow-boundary.test.ts(57 passed)npx vitest run --project e2e-support --maxWorkers=4reached 1,278 passing tests and one unrelated 5-second timeout; that exact file passed alone (31 passed). Draft CI remains authoritative.npm run docsbuilds without warnings (doc changes only)Signed-off-by: Charan Jagwani cjagwani@nvidia.com