feat(ci): coderabbit E2E recommendations + selective nightly dispatch - #2615
Merged
jyaunches merged 4 commits intoApr 28, 2026
Merged
Conversation
…#2564) Add path_instructions to .coderabbit.yaml that map sensitive file changes to recommended nightly E2E jobs. CodeRabbit surfaces these as review comments on PRs touching entrypoint scripts, Dockerfile, proxy rewrite, onboard logic, deploy, shields, Hermes, and network policies. Add a `jobs` input to the nightly-e2e.yaml workflow_dispatch trigger so maintainers can run a subset of nightly jobs on any branch: gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=sandbox-survival-e2e,sandbox-operations-e2e Each of the 18 E2E jobs gets a conditional that checks the input, so unselected jobs are skipped. Scheduled runs and empty `jobs` input still run everything. The notify-on-failure job is unaffected (skipped jobs produce result 'skipped', not 'failure'). Add test/validate-e2e-coverage.test.ts to cross-validate: - Every job name in CodeRabbit instructions exists in nightly-e2e.yaml - Every path glob in CodeRabbit instructions matches at least one file - Every nightly job has the selective dispatch guard in its if: condition - Advisory warning for nightly jobs with no CodeRabbit coverage Closes #2564 (Phases 1-3)
Contributor
📝 WalkthroughWalkthroughAdds CodeRabbit path-to-job review instructions, a workflow_dispatch Changes
Sequence Diagram(s)sequenceDiagram
participant PR as "Contributor (PR)"
participant CodeRabbit as "CodeRabbit"
participant Maintainer as "Maintainer"
participant GH as "GitHub Actions"
participant Jobs as "Nightly E2E jobs"
PR->>CodeRabbit: Pushes changes touching mapped paths
CodeRabbit-->>PR: Posts review with recommended `*-e2e` jobs and `gh workflow run ... -f jobs=...`
Maintainer->>GH: Runs `gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=jobA,jobB`
GH->>GH: Evaluate workflow_dispatch.inputs.jobs and per-job `if:` guards
GH-->>Jobs: Execute only matching E2E jobs (or all if inputs.jobs empty)
Jobs-->>GH: Return results
GH-->>Maintainer: report-to-pr posts summary comment on PR (for workflow_dispatch)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/nightly-e2e.yaml:
- Around line 62-66: Update each E2E job guard to prevent substring matches by
wrapping inputs.jobs with delimiters and checking for the delimited job token;
replace occurrences like contains(inputs.jobs, 'cloud-e2e') with
contains(concat(',', inputs.jobs, ','), ',cloud-e2e,') for all listed jobs
(cloud-e2e, messaging-providers-e2e, token-rotation-e2e, sandbox-survival-e2e,
hermes-e2e, skip-permissions-e2e, sandbox-operations-e2e, inference-routing-e2e,
network-policy-e2e, deployment-services-e2e, diagnostics-e2e,
snapshot-commands-e2e, shields-config-e2e, rebuild-openclaw-e2e,
upgrade-stale-sandbox-e2e, rebuild-hermes-e2e, overlayfs-autofix-e2e, gpu-e2e)
so each if condition uses contains(concat(',', inputs.jobs, ','),
',<job-name>,') instead of the plain contains(...) check.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 7e3e56bd-8997-480b-9601-fcea3fecb78e
📒 Files selected for processing (3)
.coderabbit.yaml.github/workflows/nightly-e2e.yamltest/validate-e2e-coverage.test.ts
…positives
CodeRabbit correctly identified that contains(inputs.jobs, 'cloud-e2e')
performs substring matching, so passing 'e2e' would match every job.
Replace all 18 job guards with:
contains(format(',{0},', inputs.jobs), ',<job-name>,')
This wraps the comma-separated input in delimiters so each job name
must match exactly between commas.
Update the cross-validation test to check for the new pattern.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
test/validate-e2e-coverage.test.ts (1)
74-75: Consider widening job-name regex to include digits.Line 74-75 currently only matches lowercase letters/hyphens. If a future job name includes digits (for example
foo2-e2e), it will be silently missed by the cross-validation.Optional hardening
- const backtickPattern = /`([a-z][-a-z]*-e2e)`/g; - const jobsArgPattern = /-f jobs=([a-z][-a-z,]*-e2e)/g; + const backtickPattern = /`([a-z][a-z0-9-]*-e2e)`/g; + const jobsArgPattern = /-f jobs=([a-z][a-z0-9,-]*-e2e)/g;🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@test/validate-e2e-coverage.test.ts` around lines 74 - 75, The current regexes backtickPattern and jobsArgPattern only allow lowercase letters and hyphens, so job names with digits (e.g., foo2-e2e) are missed; update both patterns to include digits in the character classes (e.g., change /`([a-z][-a-z]*-e2e)`/g to /`([a-z0-9][-a-z0-9]*-e2e)`/g and /-f jobs=([a-z][-a-z,]*-e2e)/g to /-f jobs=([a-z0-9][-a-z0-9,]*-e2e)/g) so digits are accepted anywhere after the first character.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@test/validate-e2e-coverage.test.ts`:
- Around line 214-217: The current validation for selective-dispatch guards only
checks that the condition string contains "inputs.jobs" and the
contains(format(...), ',${name},') fragment (variables `condition` and `name`),
which allows conditions that omit the required checks for dispatch vs
scheduled/default behavior; update the test in validate-e2e-coverage.test.ts so
the assertion also requires the condition to include the guard
"github.event_name != 'workflow_dispatch'" and the default/scheduled branch
check "inputs.jobs == ''" (or equivalent exact substrings used in your
workflows), so the test enforces presence of all four pieces: "inputs.jobs", the
contains(...) fragment, "github.event_name != 'workflow_dispatch'", and
"inputs.jobs == ''".
---
Nitpick comments:
In `@test/validate-e2e-coverage.test.ts`:
- Around line 74-75: The current regexes backtickPattern and jobsArgPattern only
allow lowercase letters and hyphens, so job names with digits (e.g., foo2-e2e)
are missed; update both patterns to include digits in the character classes
(e.g., change /`([a-z][-a-z]*-e2e)`/g to /`([a-z0-9][-a-z0-9]*-e2e)`/g and /-f
jobs=([a-z][-a-z,]*-e2e)/g to /-f jobs=([a-z0-9][-a-z0-9,]*-e2e)/g) so digits
are accepted anywhere after the first character.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d8106263-8c48-421b-b036-99d0e7d79028
📒 Files selected for processing (2)
.github/workflows/nightly-e2e.yamltest/validate-e2e-coverage.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/workflows/nightly-e2e.yaml
Comment on lines
+214
to
+217
| if ( | ||
| !condition.includes("inputs.jobs") || | ||
| !condition.includes(`contains(format(',{0},', inputs.jobs), ',${name},')`) | ||
| ) { |
Contributor
There was a problem hiding this comment.
Selective-dispatch guard validation is incomplete.
On Line 214-217, the check only requires inputs.jobs + contains(...). A job condition could still pass this test while breaking scheduled/default behavior if it omits github.event_name != 'workflow_dispatch' or inputs.jobs == ''.
Proposed tightening
- if (
- !condition.includes("inputs.jobs") ||
- !condition.includes(`contains(format(',{0},', inputs.jobs), ',${name},')`)
- ) {
+ const hasDispatchBypass = condition.includes(
+ "github.event_name != 'workflow_dispatch'",
+ );
+ const hasEmptySelectionBypass = condition.includes("inputs.jobs == ''");
+ const hasExactJobMatch = condition.includes(
+ `contains(format(',{0},', inputs.jobs), ',${name},')`,
+ );
+ if (!(hasDispatchBypass && hasEmptySelectionBypass && hasExactJobMatch)) {
missing.push(name);
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if ( | |
| !condition.includes("inputs.jobs") || | |
| !condition.includes(`contains(format(',{0},', inputs.jobs), ',${name},')`) | |
| ) { | |
| const hasDispatchBypass = condition.includes( | |
| "github.event_name != 'workflow_dispatch'", | |
| ); | |
| const hasEmptySelectionBypass = condition.includes("inputs.jobs == ''"); | |
| const hasExactJobMatch = condition.includes( | |
| `contains(format(',{0},', inputs.jobs), ',${name},')`, | |
| ); | |
| if (!(hasDispatchBypass && hasEmptySelectionBypass && hasExactJobMatch)) { |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@test/validate-e2e-coverage.test.ts` around lines 214 - 217, The current
validation for selective-dispatch guards only checks that the condition string
contains "inputs.jobs" and the contains(format(...), ',${name},') fragment
(variables `condition` and `name`), which allows conditions that omit the
required checks for dispatch vs scheduled/default behavior; update the test in
validate-e2e-coverage.test.ts so the assertion also requires the condition to
include the guard "github.event_name != 'workflow_dispatch'" and the
default/scheduled branch check "inputs.jobs == ''" (or equivalent exact
substrings used in your workflows), so the test enforces presence of all four
pieces: "inputs.jobs", the contains(...) fragment, "github.event_name !=
'workflow_dispatch'", and "inputs.jobs == ''".
Add report-to-pr job to nightly-e2e.yaml. When the workflow is triggered via workflow_dispatch on a branch with an open PR, the job posts a comment with: - Which jobs were requested - A results table (pass/fail/skipped for every job) - Direct link to the Actions run - Failed job callout with artifacts link Only runs on workflow_dispatch (not nightly schedule). Silently skips if no open PR exists for the branch. Exclude report-to-pr from the cross-validation test's E2E job list (same as notify-on-failure — infrastructure, not a test job).
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
test/validate-e2e-coverage.test.ts (1)
216-219:⚠️ Potential issue | 🟠 MajorSelective-dispatch guard validation is still incomplete.
This check can pass even if a job drops required dispatch/default guard clauses. Require all key fragments, not just
inputs.jobs+ exact-tokencontains(...).Suggested fix
- if ( - !condition.includes("inputs.jobs") || - !condition.includes(`contains(format(',{0},', inputs.jobs), ',${name},')`) - ) { + const hasDispatchBypass = condition.includes( + "github.event_name != 'workflow_dispatch'", + ); + const hasEmptySelectionBypass = condition.includes("inputs.jobs == ''"); + const hasInputsRef = condition.includes("inputs.jobs"); + const hasExactJobMatch = condition.includes( + `contains(format(',{0},', inputs.jobs), ',${name},')`, + ); + if ( + !( + hasDispatchBypass && + hasEmptySelectionBypass && + hasInputsRef && + hasExactJobMatch + ) + ) { missing.push(name); }As per coding guidelines: for selective nightly dispatch, each E2E job guard must preserve scheduled/empty-input behavior and exact-token matching.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@test/validate-e2e-coverage.test.ts` around lines 216 - 219, The current guard check on the condition string only ensures presence of "inputs.jobs" and the contains(...) token, which can miss other required fragments; update the validation in validate-e2e-coverage.test.ts to assert that the condition string contains all required guard fragments (e.g., "github.event_name == 'schedule'" or equivalent scheduled check, the exact-token contains(format(',{0},', inputs.jobs), ',${name},') match, and the empty-input/default-dispatch clause that preserves scheduled/empty-input behavior) rather than just two fragments. Locate the variable/logic using condition (the if that checks condition.includes(...)) and add explicit assertions for each fragment/token required for selective-dispatch guards so any job that drops required clauses will fail the test.
🧹 Nitpick comments (1)
test/validate-e2e-coverage.test.ts (1)
186-200: Make uncovered nightly-job coverage an enforceable assertion (not warning-only).This currently always passes, so new nightly jobs can drift without required
.coderabbit.yamlmapping. Consider asserting on uncovered jobs (with an explicit allowlist for intentional exceptions).Suggested tightening
- const uncovered = nightlyJobs.filter((name) => !referencedJobs.has(name)); - // This is a warning-level check: some jobs (e.g., diagnostics-e2e, - // upgrade-stale-sandbox-e2e) may intentionally lack path-based - // recommendations. We still flag them so maintainers can decide. - if (uncovered.length > 0) { - console.warn( - `⚠️ Nightly E2E jobs with no CodeRabbit path_instructions coverage: ` + - `${uncovered.join(", ")}. ` + - `Consider adding path_instructions entries in .coderabbit.yaml ` + - `for the source files these jobs exercise.`, - ); - } - // Intentionally does not fail — this is advisory. - expect(true).toBe(true); + const allowedUncovered = new Set<string>([ + "diagnostics-e2e", + "upgrade-stale-sandbox-e2e", + ]); + const uncovered = nightlyJobs.filter( + (name) => !referencedJobs.has(name) && !allowedUncovered.has(name), + ); + expect( + uncovered, + `Nightly E2E jobs missing CodeRabbit path_instructions coverage: ` + + `${uncovered.join(", ")}.`, + ).toEqual([]);As per coding guidelines: “If a new E2E job is added, verify a corresponding
path_instructionsentry exists in.coderabbit.yaml… The cross-validation test … checks this automatically.”🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@test/validate-e2e-coverage.test.ts` around lines 186 - 200, The test currently treats uncovered nightlyJobs as a warning; change it to fail unless jobs are explicitly allowlisted: compute uncovered = nightlyJobs.filter(name => !referencedJobs.has(name) && !allowedMissing.has(name)) (introduce an allowlist/Set like allowedMissing for intentional exceptions), then replace the no-op expect with an assertion such as expect(uncovered).toHaveLength(0) or expect(uncovered).toEqual([]) and include a clear failure message listing uncovered so CI enforces adding path_instructions entries for new nightly jobs; update the test block titled "every nightly E2E job has at least one CodeRabbit path_instructions entry" accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/nightly-e2e.yaml:
- Line 785: The workflow currently injects inputs.jobs directly into a JS string
(const requestedJobs = '${{ inputs.jobs }}'), which can break if the input
contains quotes; change this to use JSON-safe interpolation so the runtime
receives a valid JS literal—replace the assignment with: const requestedJobs =
${{ toJson(inputs.jobs) }}; (update the github-script step that defines
requestedJobs) so inputs.jobs is safely serialized into JavaScript.
---
Duplicate comments:
In `@test/validate-e2e-coverage.test.ts`:
- Around line 216-219: The current guard check on the condition string only
ensures presence of "inputs.jobs" and the contains(...) token, which can miss
other required fragments; update the validation in validate-e2e-coverage.test.ts
to assert that the condition string contains all required guard fragments (e.g.,
"github.event_name == 'schedule'" or equivalent scheduled check, the exact-token
contains(format(',{0},', inputs.jobs), ',${name},') match, and the
empty-input/default-dispatch clause that preserves scheduled/empty-input
behavior) rather than just two fragments. Locate the variable/logic using
condition (the if that checks condition.includes(...)) and add explicit
assertions for each fragment/token required for selective-dispatch guards so any
job that drops required clauses will fail the test.
---
Nitpick comments:
In `@test/validate-e2e-coverage.test.ts`:
- Around line 186-200: The test currently treats uncovered nightlyJobs as a
warning; change it to fail unless jobs are explicitly allowlisted: compute
uncovered = nightlyJobs.filter(name => !referencedJobs.has(name) &&
!allowedMissing.has(name)) (introduce an allowlist/Set like allowedMissing for
intentional exceptions), then replace the no-op expect with an assertion such as
expect(uncovered).toHaveLength(0) or expect(uncovered).toEqual([]) and include a
clear failure message listing uncovered so CI enforces adding path_instructions
entries for new nightly jobs; update the test block titled "every nightly E2E
job has at least one CodeRabbit path_instructions entry" accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8cb2a1a6-1d32-49ab-b530-bcc164f23f71
📒 Files selected for processing (2)
.github/workflows/nightly-e2e.yamltest/validate-e2e-coverage.test.ts
| const needs = ${{ toJSON(needs) }}; | ||
| const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`; | ||
| const branch = context.ref.replace('refs/heads/', ''); | ||
| const requestedJobs = '${{ inputs.jobs }}'; |
Contributor
There was a problem hiding this comment.
Use JSON-safe interpolation for inputs.jobs in github-script.
Line 785 injects the workflow input directly into a single-quoted JS string. A quote in input can break the script and skip PR reporting.
Suggested fix
- const requestedJobs = '${{ inputs.jobs }}';
+ const requestedJobs = ${{ toJSON(inputs.jobs) }} || "";🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/nightly-e2e.yaml at line 785, The workflow currently
injects inputs.jobs directly into a JS string (const requestedJobs = '${{
inputs.jobs }}'), which can break if the input contains quotes; change this to
use JSON-safe interpolation so the runtime receives a valid JS literal—replace
the assignment with: const requestedJobs = ${{ toJson(inputs.jobs) }}; (update
the github-script step that defines requestedJobs) so inputs.jobs is safely
serialized into JavaScript.
cv
approved these changes
Apr 28, 2026
jyaunches
changed the title
Contributor
Selective E2E Results — ✅ All requested jobs passedRun: 25058486980
|
wscurran
added
enhancement
…eRabbit review - Resolve merge conflict: keep main's runner/timeout for gpu-e2e, retain selective dispatch guard - Add selective dispatch guard to new gpu-double-onboard-e2e job from main - Tighten guard validation test to check all three clauses - Use toJSON(inputs.jobs) for safe JS interpolation in report-to-pr
cv
approved these changes
Apr 28, 2026
ericksoa
added a commit
that referenced
this pull request
Apr 28, 2026
Brings in the selective nightly-e2e dispatch infrastructure from #2615 (adds workflow_dispatch.inputs.jobs filter + per-job guard) and the test/validate-e2e-coverage.test.ts cross-check that asserts every nightly job carries the canonical guard. Updates the issue-2478-crash-loop-recovery-e2e job to use the same guard pattern as every other job, and adds it to the inputs.jobs description "Valid:" list. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
10 tasks
DemianHeyGen
pushed a commit
to DemianHeyGen/NemoClaw
that referenced
this pull request
Apr 30, 2026
…NVIDIA#2615) ## Summary Add automated E2E test recommendations to PR reviews and selective job dispatch to the nightly E2E workflow. Closes NVIDIA#2564 (Phases 1–3). ## What changed ### 1. CodeRabbit `path_instructions` for E2E recommendations (`.coderabbit.yaml`) 15 new `path_instructions` entries map sensitive file paths to the nightly E2E jobs that exercise them. When a PR touches a mapped path, CodeRabbit posts a review comment recommending specific jobs and a copy-pasteable `gh workflow run` command. | Path Pattern | Recommended Jobs | |-------------|-----------------| | `scripts/nemoclaw-start.sh`, `scripts/lib/sandbox-init.sh` | `sandbox-survival-e2e`, `sandbox-operations-e2e`, `cloud-e2e` | | `Dockerfile`, `Dockerfile.base` | `cloud-e2e`, `sandbox-survival-e2e`, `hermes-e2e`, `rebuild-openclaw-e2e` | | `nemoclaw-blueprint/scripts/http-proxy-fix.js` | `cloud-e2e`, `inference-routing-e2e` | | `src/lib/onboard.ts` | `cloud-e2e`, `sandbox-operations-e2e`, `rebuild-openclaw-e2e` | | `src/nemoclaw.ts` | `sandbox-survival-e2e`, `sandbox-operations-e2e`, `skip-permissions-e2e` | | `src/lib/cluster-image-patch.ts`, `src/lib/preflight.ts` | `overlayfs-autofix-e2e` | | `src/lib/deploy.ts` | `deployment-services-e2e` | | `src/lib/sandbox-state.ts` | `snapshot-commands-e2e`, `rebuild-openclaw-e2e` | | `src/lib/shields*.ts` | `shields-config-e2e` | | `agents/hermes/**` | `hermes-e2e`, `rebuild-hermes-e2e` | | `nemoclaw-blueprint/policies/**` | `network-policy-e2e`, `skip-permissions-e2e` | | `.github/workflows/nightly-e2e.yaml` | Reminds to add CodeRabbit coverage for new jobs | ### 2. Selective job dispatch (`nightly-e2e.yaml`) Added a `jobs` input to `workflow_dispatch` so maintainers can run a subset of nightly jobs on any branch: ``` gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=sandbox-survival-e2e,sandbox-operations-e2e ``` - All 18 E2E jobs get a conditional guard: unselected jobs are skipped - Empty `jobs` input (or scheduled runs) still runs everything - `notify-on-failure` is unaffected: skipped jobs produce `result: 'skipped'`, not `'failure'` ### 3. Cross-validation test (`test/validate-e2e-coverage.test.ts`) Keeps the mapping up to date as files and jobs evolve: | Assertion | What it catches | |-----------|----------------| | Job names in CodeRabbit match `nightly-e2e.yaml` | Renamed/removed jobs | | Path globs match at least one file on disk | Renamed/deleted source files | | Every nightly job has selective dispatch guard | New jobs added without the `if:` pattern | | Advisory: nightly jobs with no CodeRabbit coverage | New jobs added without `path_instructions` | ## Validation - [x] All 4 cross-validation tests pass locally - [x] Existing `validate-config-schemas` tests still pass - [x] Selective dispatch validated: [run 25052625486](https://github.com/NVIDIA/NemoClaw/actions/runs/25052625486) — triggered with `-f jobs=diagnostics-e2e`, 17/18 jobs correctly skipped - [x] `notify-on-failure` does not false-alarm on selective run — [run 25052625486](https://github.com/NVIDIA/NemoClaw/actions/runs/25052625486) confirmed: `notify-on-failure` was skipped (not triggered) - [ ] CodeRabbit posts recommendations on a PR touching a mapped file (post-merge validation) ## Context - Issue: NVIDIA#2564 - Weekend incident: NVIDIA#2471, NVIDIA#2472, NVIDIA#2482, NVIDIA#2490 - E2E strategy: `cloud-experimental-e2e` removal in NVIDIA#2472 left a coverage gap that would have been flagged by these recommendations <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Expanded review automation to map sensitive paths to targeted nightly E2E jobs and inject instructions for running relevant subsets. * Added manual workflow dispatch allowing selective E2E job execution via a jobs input. * **New Features** * Added a reporting step that, on manual runs, posts a PR comment summarizing passed/failed/skipped E2E jobs. * **Tests** * Added a validation suite that cross-checks review-to-workflow mappings and dispatch guards, warning on uncovered jobs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> ### 4. Substring match fix (`nightly-e2e.yaml`) CodeRabbit review correctly identified that `contains(inputs.jobs, 'cloud-e2e')` performs substring matching — e.g., passing `jobs=e2e` would match every job. All 18 job guards now use delimiter-wrapping: ```yaml contains(format(',{0},', inputs.jobs), ',<job-name>,') ``` This ensures exact token matching within the comma-separated input. The cross-validation test was updated to enforce the new pattern.
7 tasks
wscurran
added
area: ci
jyaunches
deleted the
issue-2564-coderabbit-e2e-recommendations-selective-dispatch
branch
June 12, 2026 13:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add automated E2E test recommendations to PR reviews and selective job dispatch to the nightly E2E workflow.
Closes #2564 (Phases 1–3).
What changed
1. CodeRabbit
path_instructionsfor E2E recommendations (.coderabbit.yaml)15 new
path_instructionsentries map sensitive file paths to the nightly E2E jobs that exercise them. When a PR touches a mapped path, CodeRabbit posts a review comment recommending specific jobs and a copy-pasteablegh workflow runcommand.scripts/nemoclaw-start.sh,scripts/lib/sandbox-init.shsandbox-survival-e2e,sandbox-operations-e2e,cloud-e2eDockerfile,Dockerfile.basecloud-e2e,sandbox-survival-e2e,hermes-e2e,rebuild-openclaw-e2enemoclaw-blueprint/scripts/http-proxy-fix.jscloud-e2e,inference-routing-e2esrc/lib/onboard.tscloud-e2e,sandbox-operations-e2e,rebuild-openclaw-e2esrc/nemoclaw.tssandbox-survival-e2e,sandbox-operations-e2e,skip-permissions-e2esrc/lib/cluster-image-patch.ts,src/lib/preflight.tsoverlayfs-autofix-e2esrc/lib/deploy.tsdeployment-services-e2esrc/lib/sandbox-state.tssnapshot-commands-e2e,rebuild-openclaw-e2esrc/lib/shields*.tsshields-config-e2eagents/hermes/**hermes-e2e,rebuild-hermes-e2enemoclaw-blueprint/policies/**network-policy-e2e,skip-permissions-e2e.github/workflows/nightly-e2e.yaml2. Selective job dispatch (
nightly-e2e.yaml)Added a
jobsinput toworkflow_dispatchso maintainers can run a subset of nightly jobs on any branch:jobsinput (or scheduled runs) still runs everythingnotify-on-failureis unaffected: skipped jobs produceresult: 'skipped', not'failure'3. Cross-validation test (
test/validate-e2e-coverage.test.ts)Keeps the mapping up to date as files and jobs evolve:
nightly-e2e.yamlif:patternpath_instructionsValidation
validate-config-schemastests still pass-f jobs=diagnostics-e2e, 17/18 jobs correctly skippednotify-on-failuredoes not false-alarm on selective run — run 25052625486 confirmed:notify-on-failurewas skipped (not triggered)Context
cloud-experimental-e2eremoval in fix(sandbox): fix non-root gateway startup and add crash safety net #2472 left a coverage gap that would have been flagged by these recommendationsSummary by CodeRabbit
Chores
New Features
Tests
4. Substring match fix (
nightly-e2e.yaml)CodeRabbit review correctly identified that
contains(inputs.jobs, 'cloud-e2e')performs substring matching — e.g., passingjobs=e2ewould match every job. All 18 job guards now use delimiter-wrapping:contains(format(',{0},', inputs.jobs), ',<job-name>,')This ensures exact token matching within the comma-separated input. The cross-validation test was updated to enforce the new pattern.