ci(e2e): add risk-based PR gate - #6662
Merged
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis change replaces post-merge risk-shadow orchestration with a required-live PR E2E gate. It adds a controller, validates correlated child workflows and evidence, updates E2E dispatch contracts, removes shadow workflows, and revises risk-plan schemas, tests, and documentation. ChangesRisk-plan and evidence contracts
Required-live controller lifecycle
Child workflow dispatch and evidence wiring
Required-live workflow coordination
Documentation and ownership migration
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant CI as CI / Pull Request
participant Controller as PR E2E Controller
participant GitHub as GitHub API
participant E2E as E2E Workflow
participant Evidence as Evidence Artifacts
CI->>Controller: trigger after completion
Controller->>GitHub: resolve PR and changed files
Controller->>GitHub: dispatch required jobs
E2E->>Evidence: upload correlated risk signals
Controller->>GitHub: poll child run
Controller->>Evidence: validate shard evidence
Controller->>GitHub: complete PR E2E check
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-6662.docs.buildwithfern.com/nemoclaw |
Contributor
E2E Advisor RecommendationRequired E2E: Dispatch hint: Full advisor summaryE2E Recommendation AdvisorBase: Required E2E
Optional E2E
New E2E recommendations
Dispatch hint
|
Contributor
E2E Target RecommendationRequired E2E targets: Dispatch required E2E targets:
Full E2E target advisor summaryE2E Target AdvisorBase: Required E2E targets
Optional E2E targets
Relevant changed files
|
Contributor
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage remains at 96%, unchanged from the TypeScript / code-coverage/cliThe overall coverage in the Show a code coverage summary of the most impacted files.
Updated |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
PR Review Advisor — Changes requestedMerge posture: Do not merge yet Action checklist
Findings index
Review findings by urgency: 0 required fixes, 1 item to resolve/justify, 0 in-scope improvements
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (4)
test/e2e/README.md (1)
11-14: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winKeep each sentence on one Markdown source line.
The changed prose wraps individual sentences across multiple physical lines, contrary to the documentation guideline. Reflow the affected sentences without changing their content.
As per coding guidelines, Markdown and MDX files must keep one sentence per line.
Also applies to: 58-115
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e/README.md` around lines 11 - 14, The affected documentation sentences in the E2E README, including the sections spanning the referenced range, are wrapped across multiple Markdown source lines. Reflow each sentence so it occupies one physical line, preserving all wording and content.Source: Coding guidelines
tools/e2e-advisor/README.md (1)
41-91: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winKeep each sentence on one Markdown source line.
The changed prose wraps individual sentences across multiple physical lines. Reflow the affected sentences to comply with the documentation guideline.
As per coding guidelines, Markdown and MDX files must keep one sentence per line.
Also applies to: 116-121, 153-154
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tools/e2e-advisor/README.md` around lines 41 - 91, Reflow the affected prose in README.md so every sentence occupies a single Markdown source line, including the sections referenced around the changed content and the additional ranges 116-121 and 153-154. Preserve the wording and paragraph structure while joining wrapped sentence fragments onto one physical line.Source: Coding guidelines
test/e2e/docs/README.md (1)
109-123: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winKeep each sentence on one Markdown source line.
The changed documentation wraps individual sentences across multiple physical lines. Reflow these sentences to comply with the repository Markdown guideline.
As per coding guidelines, Markdown and MDX files must keep one sentence per line.
Also applies to: 136-145
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e/docs/README.md` around lines 109 - 123, Reflow the affected documentation in the README so each complete sentence occupies a single Markdown source line. Update the wrapped sentences in the sections around the shown diff and the additionally referenced section, preserving wording and paragraph structure while ensuring no sentence spans multiple physical lines.Source: Coding guidelines
tools/pr-review-advisor/README.md (1)
59-63: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winKeep each sentence on one Markdown source line.
The changed documentation wraps individual sentences across multiple physical lines. Reflow these sentences to comply with the repository Markdown guideline.
As per coding guidelines, Markdown and MDX files must keep one sentence per line.
Also applies to: 87-88
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tools/pr-review-advisor/README.md` around lines 59 - 63, Reflow the affected documentation in the README so each sentence occupies a single Markdown source line. Update the wrapped sentences in the shown section and the corresponding lines around the referenced additional location, preserving wording and paragraph structure.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/required-live-e2e.yaml:
- Around line 83-95: Prevent shell injection in the “start” step by passing
github.event.workflow_run.head_branch through the step’s env as HEAD_BRANCH and
referencing it safely as "$HEAD_BRANCH" in the command. Update the corresponding
assertion in required-live-workflow.test.ts to expect the environment-based
reference.
In `@test/e2e/README.md`:
- Around line 79-80: Update the dispatch contract wording in the E2E guide to
state that selective job dispatch must be performed without the `targets` input,
replacing the ambiguous “empty target fan-out” phrasing.
In `@tools/e2e-advisor/README.md`:
- Around line 61-62: Clarify the required-live dispatch prerequisites in the
README sentence: state that it uses selective jobs with no `targets` input,
rather than describing this as an “empty target fan-out.”
---
Nitpick comments:
In `@test/e2e/docs/README.md`:
- Around line 109-123: Reflow the affected documentation in the README so each
complete sentence occupies a single Markdown source line. Update the wrapped
sentences in the sections around the shown diff and the additionally referenced
section, preserving wording and paragraph structure while ensuring no sentence
spans multiple physical lines.
In `@test/e2e/README.md`:
- Around line 11-14: The affected documentation sentences in the E2E README,
including the sections spanning the referenced range, are wrapped across
multiple Markdown source lines. Reflow each sentence so it occupies one physical
line, preserving all wording and content.
In `@tools/e2e-advisor/README.md`:
- Around line 41-91: Reflow the affected prose in README.md so every sentence
occupies a single Markdown source line, including the sections referenced around
the changed content and the additional ranges 116-121 and 153-154. Preserve the
wording and paragraph structure while joining wrapped sentence fragments onto
one physical line.
In `@tools/pr-review-advisor/README.md`:
- Around line 59-63: Reflow the affected documentation in the README so each
sentence occupies a single Markdown source line. Update the wrapped sentences in
the shown section and the corresponding lines around the referenced additional
location, preserving wording and paragraph structure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3152a77a-3a3b-4187-9490-667cde078c34
📒 Files selected for processing (25)
.github/workflows/e2e.yaml.github/workflows/post-merge-e2e-risk-gate-shadow.yaml.github/workflows/required-live-e2e.yamldocs/about/release-notes.mdxtest/e2e-advisor-targets.test.tstest/e2e-private-file.test.tstest/e2e-risk-signal-reporter.test.tstest/e2e/README.mdtest/e2e/docs/README.mdtest/e2e/risk-signal-reporter.tstest/e2e/support/e2e-operations-workflow-boundary.test.tstest/post-merge-e2e-risk-gate-workflow.test.tstest/post-merge-e2e-risk-gate.test.tstest/pr-risk-plan.test.tstest/required-live-workflow.test.tstest/required-live.test.tstools/advisors/risk-plan.mtstools/e2e-advisor/README.mdtools/e2e-advisor/post-merge-risk-gate.mtstools/e2e/operations-workflow-boundary.mtstools/e2e/private-file.tstools/e2e/required-live.mtstools/e2e/risk-signal.tstools/pr-review-advisor/README.mdtools/pr-review-advisor/analyze.mts
💤 Files with no reviewable changes (5)
- test/post-merge-e2e-risk-gate.test.ts
- .github/workflows/post-merge-e2e-risk-gate-shadow.yaml
- test/post-merge-e2e-risk-gate-workflow.test.ts
- tools/e2e-advisor/post-merge-risk-gate.mts
- tools/pr-review-advisor/analyze.mts
Collaborator
Author
Automated review follow-up
|
Contributor
PR Review Advisor (Nemotron Ultra) — Changes requestedMerge posture: Do not merge yet Action checklist
Findings index
🚨 Required before mergeAddress these before merging unless a maintainer explicitly overrides the advisor with rationale.
|
19 tasks
cv
changed the title
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
.github/workflows/pr-e2e-gate.yaml (1)
202-223: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winRoute step outputs through
envinstead of interpolating intorun:(defense-in-depth).The
finishandabandonsteps inline${{ steps.*.outputs.* }}directly into the shell program, which zizmor flags as template-injection. These specific outputs are currently safe (state_hash/check_id/run_idpass the controller's strictappendOutputallowlist, andwork_diris amktemppath), so this is not exploitable today. However, thestart/waitsteps already use theenv-then-"$VAR"pattern; matching it here keeps the workflow consistent, silences the SAST finding, and future-proofs against a later output on the same path becoming attacker-influenced. The same pattern applies towork_dirin theevidence(Line 196) and cleanup (Line 227) steps.As per path instructions: "pass untrusted values as data rather than interpolating them into shell programs."
🔒 Proposed hardening for the finish/abandon steps
- id: finish name: Verify evidence if: ${{ always() && steps.start.outputs.dispatched == 'true' }} env: GITHUB_TOKEN: ${{ github.token }} + WORK_DIR: ${{ steps.workspace.outputs.work_dir }} + STATE_HASH: ${{ steps.start.outputs.state_hash }} + CHECK_ID: ${{ steps.start.outputs.check_id }} + RUN_ID: ${{ steps.start.outputs.run_id }} run: >- node --experimental-strip-types tools/e2e/pr-e2e-gate.mts --mode finish - --work-dir "${{ steps.workspace.outputs.work_dir }}" - --state-hash "${{ steps.start.outputs.state_hash }}" - --check-id "${{ steps.start.outputs.check_id }}" - --run-id "${{ steps.start.outputs.run_id }}" + --work-dir "$WORK_DIR" + --state-hash "$STATE_HASH" + --check-id "$CHECK_ID" + --run-id "$RUN_ID" - name: Close incomplete check if: ${{ always() && steps.start.outputs.check_id != '' && steps.start.outputs.finalized != 'true' && steps.finish.outputs.finalized != 'true' }} env: GITHUB_TOKEN: ${{ github.token }} + CHECK_ID: ${{ steps.start.outputs.check_id }} + RUN_ID: ${{ steps.start.outputs.run_id }} run: >- node --experimental-strip-types tools/e2e/pr-e2e-gate.mts --mode abandon - --check-id "${{ steps.start.outputs.check_id }}" - --run-id "${{ steps.start.outputs.run_id }}" + --check-id "$CHECK_ID" + --run-id "$RUN_ID"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/pr-e2e-gate.yaml around lines 202 - 223, Route all GitHub step outputs used by the “Verify evidence”, “Close incomplete check”, “evidence”, and cleanup steps through step-level env variables instead of interpolating them directly into run commands. Update the scripts to reference those values as quoted shell environment variables, including work_dir, state_hash, check_id, and run_id, matching the existing start/wait pattern.Sources: Path instructions, Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In @.github/workflows/pr-e2e-gate.yaml:
- Around line 202-223: Route all GitHub step outputs used by the “Verify
evidence”, “Close incomplete check”, “evidence”, and cleanup steps through
step-level env variables instead of interpolating them directly into run
commands. Update the scripts to reference those values as quoted shell
environment variables, including work_dir, state_hash, check_id, and run_id,
matching the existing start/wait pattern.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: be139e3e-70aa-45dd-9efc-8d97270f9ef9
📒 Files selected for processing (13)
.github/workflows/e2e.yaml.github/workflows/pr-e2e-gate.yamldocs/about/release-notes.mdxtest/e2e-risk-signal-reporter.test.tstest/e2e/README.mdtest/e2e/docs/README.mdtest/e2e/support/e2e-operations-workflow-boundary.test.tstest/pr-e2e-gate-workflow.test.tstest/pr-e2e-gate.test.tstools/e2e-advisor/README.mdtools/e2e/operations-workflow-boundary.mtstools/e2e/pr-e2e-gate.mtstools/pr-review-advisor/README.md
✅ Files skipped from review due to trivial changes (2)
- docs/about/release-notes.mdx
- tools/pr-review-advisor/README.md
🚧 Files skipped from review as they are similar to previous changes (3)
- test/e2e/support/e2e-operations-workflow-boundary.test.ts
- test/e2e-risk-signal-reporter.test.ts
- .github/workflows/e2e.yaml
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
21 tasks
cv
added a commit
that referenced
this pull request
Jul 11, 2026
<!-- markdownlint-disable MD041 --> ## Summary Follow-up to #6662, this replaces repeated credential-free E2E jobs with source discovery and one shared Ubuntu E2E job. Tests opt in beside their code with `e2e/credential-free`; the E2E workflow owns the runner, setup, permissions, secrets, timeout, and artifact policy. ## Changes - Discover tagged tests from their real Vitest projects and derive test IDs from unique filenames, with no separate test catalog or manually maintained matrix. - Run five credential-free tests in the `shared-e2e` matrix job, including the previously unwired Ubuntu repository CLI smoke test. - Move selector and matrix planning from inline workflow Bash into tested TypeScript while preserving `jobs`/`targets` selectors, PR reporting, scorecard inputs, and controller evidence. - Keep the temporary four-key shell output required by the current base E2E workflow. It discovers that workflow's actual job IDs, excludes selectors the base cannot run, and can be deleted once this workflow is on `main`. - Reduce `.github/workflows/e2e.yaml` from 5,430 to 5,334 lines and document the contributor contract in `test/e2e/README.md`. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent diff and workflow-contract reviews found no credential exposure, path injection, untrusted runner selection, or unresolved Actions contract mismatch; boundary tests pin the credential-free shared job's no-secret setup and artifact contracts. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run --project e2e-support` (113 files, 910 tests); PR gate/advisor/mock-parity integration suites (52 tests); `npm run test:projects:check`; `npm run typecheck:cli`; real `origin/main...HEAD` mock-parity check; exact current-base shell/matrix compatibility simulation - [x] Applicable broad gate passed — earlier PR head: `env -u SSH_CONNECTION -u SSH_CLIENT -u SSH_TTY npm test` (1,388 files, 15,964 tests passed; 39 skipped; 1 todo); current terminology change set reran the complete E2E-support project - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — passed with 0 errors and 2 existing Fern warnings - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) — no `docs/` pages changed - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added discovery and execution of “credential-free” E2E tests via a shared workflow job. * Added a dedicated workflow planner that builds a live test matrix plus a credential-free test matrix from typed selectors. * Improved PR reporting to include per-test results (including handling for unavailable/unknown results). * **Bug Fixes** * Improved selector parsing/selection behavior for shared E2E routing and dispatch targeting. * Prevented metadata-only changes from being treated as relevant for mock-parity checks. * **Documentation** * Added documentation for credential-free tests, tagging, discovery rules, and local inspection commands. * **Tests** * Extended E2E coverage for selector normalization, credential-free discovery, workflow boundaries, and PR reporting scenarios (including unknown/missing results). <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
11 tasks
Hadar301
pushed a commit
to Hadar301/NemoClaw-OpenShift
that referenced
this pull request
Jul 12, 2026
<!-- markdownlint-disable MD041 --> ## Summary This PR replaces separate advisory and post-merge control paths with one deterministic PR E2E gate. The gate selects jobs from the checked-in risk policy, verifies complete evidence for the PR head commit, and keeps model output outside merge authority; the final refactor names each workflow, run, and controller by its actual role. ## Changes - add `E2E / PR Gate` with one controller for planning, dispatch, cancellation, evidence verification, and check completion - reject fork events before trusted check or API mutations - reduce the child `e2e.yaml` dispatch contract to the PR commit, selected jobs, plan hash, and correlation ID - require every policy-selected job shard to produce bound evidence and fail the check for every incomplete outcome - rename the workflow, controller, tests, artifacts, symbols, steps, and diagnostics without aliases or compatibility paths - consolidate the lifecycle documentation in `test/e2e/README.md` and keep advisor documentation focused on its local relationship ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check exactly one tests line and one docs line. Check other lines when applicable. Add every requested justification or approval reference. --> - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent code and test reviews found no remaining actionable issues; security boundaries remain covered by workflow tests - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification <!-- Check each applicable item only when supported by the requested evidence. Run targeted tests once per relevant change set and rerun after later edits or hook autofixes that can affect the tested behavior. Do not rerun hook-covered checks. --> - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 43 integration tests, 24 E2E-support tests, `npm run typecheck:cli`, and `npm run test:projects:check` - [x] Applicable broad gate passed — `npm test` passed earlier in this PR with 15,850 tests; the current naming refactor was reverified with the focused suites above - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — passed with 0 errors and 2 pre-existing Fern warnings - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added a required PR E2E gate that builds a deterministic risk plan, dispatches only the needed E2E jobs, and blocks until all evidence signals are complete and passing. - Introduced controller-driven validation for manual dispatches, plus automatic cancellation of superseded/invalid PR E2E runs and updated PR reporting/scorecard gating. - **Documentation** - Updated E2E and release documentation to describe the PR gate flow and the new “required-live” terminology/behavior. - **Tests** - Expanded workflow contract, evidence/shard routing, and risk-plan determinism coverage; updated reporter and prompt-context expectations for the new signal schema. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Hadar301
pushed a commit
to Hadar301/NemoClaw-OpenShift
that referenced
this pull request
Jul 12, 2026
<!-- markdownlint-disable MD041 --> ## Summary Follow-up to NVIDIA#6662, this replaces repeated credential-free E2E jobs with source discovery and one shared Ubuntu E2E job. Tests opt in beside their code with `e2e/credential-free`; the E2E workflow owns the runner, setup, permissions, secrets, timeout, and artifact policy. ## Changes - Discover tagged tests from their real Vitest projects and derive test IDs from unique filenames, with no separate test catalog or manually maintained matrix. - Run five credential-free tests in the `shared-e2e` matrix job, including the previously unwired Ubuntu repository CLI smoke test. - Move selector and matrix planning from inline workflow Bash into tested TypeScript while preserving `jobs`/`targets` selectors, PR reporting, scorecard inputs, and controller evidence. - Keep the temporary four-key shell output required by the current base E2E workflow. It discovers that workflow's actual job IDs, excludes selectors the base cannot run, and can be deleted once this workflow is on `main`. - Reduce `.github/workflows/e2e.yaml` from 5,430 to 5,334 lines and document the contributor contract in `test/e2e/README.md`. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent diff and workflow-contract reviews found no credential exposure, path injection, untrusted runner selection, or unresolved Actions contract mismatch; boundary tests pin the credential-free shared job's no-secret setup and artifact contracts. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run --project e2e-support` (113 files, 910 tests); PR gate/advisor/mock-parity integration suites (52 tests); `npm run test:projects:check`; `npm run typecheck:cli`; real `origin/main...HEAD` mock-parity check; exact current-base shell/matrix compatibility simulation - [x] Applicable broad gate passed — earlier PR head: `env -u SSH_CONNECTION -u SSH_CLIENT -u SSH_TTY npm test` (1,388 files, 15,964 tests passed; 39 skipped; 1 todo); current terminology change set reran the complete E2E-support project - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — passed with 0 errors and 2 existing Fern warnings - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) — no `docs/` pages changed - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added discovery and execution of “credential-free” E2E tests via a shared workflow job. * Added a dedicated workflow planner that builds a live test matrix plus a credential-free test matrix from typed selectors. * Improved PR reporting to include per-test results (including handling for unavailable/unknown results). * **Bug Fixes** * Improved selector parsing/selection behavior for shared E2E routing and dispatch targeting. * Prevented metadata-only changes from being treated as relevant for mock-parity checks. * **Documentation** * Added documentation for credential-free tests, tagging, discovery rules, and local inspection commands. * **Tests** * Extended E2E coverage for selector normalization, credential-free discovery, workflow boundaries, and PR reporting scenarios (including unknown/missing results). <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
This was referenced Jul 20, 2026
wscurran
added
area: ci
23 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR replaces separate advisory and post-merge control paths with one deterministic PR E2E gate. The gate selects jobs from the checked-in risk policy, verifies complete evidence for the PR head commit, and keeps model output outside merge authority; the final refactor names each workflow, run, and controller by its actual role.
Changes
E2E / PR Gatewith one controller for planning, dispatch, cancellation, evidence verification, and check completione2e.yamldispatch contract to the PR commit, selected jobs, plan hash, and correlation IDtest/e2e/README.mdand keep advisor documentation focused on its local relationshipType of Change
Quality Gates
Verification
Verifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailablenpm run typecheck:cli, andnpm run test:projects:checknpm testpassed earlier in this PR with 15,850 tests; the current naming refactor was reverified with the focused suites abovenpm run docsbuilds without warnings (doc changes only) — passed with 0 errors and 2 pre-existing Fern warningsSigned-off-by: Carlos Villela cvillela@nvidia.com
Summary by CodeRabbit
New Features
Documentation
Tests