From 9ce7faa309e8896b60fc2d1a31212f94923ef6ea Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Thu, 16 Jul 2026 13:34:39 -0700 Subject: [PATCH 1/7] fix(ci): run selected typed targets in PR E2E Signed-off-by: Carlos Villela --- .../nemoclaw-maintainer-day/MERGE-GATE.md | 6 +- .github/workflows/e2e.yaml | 19 +- test/e2e/README.md | 65 ++++--- test/e2e/support/e2e-workflow.test.ts | 44 ++++- test/e2e/support/workflow-plan.test.ts | 15 +- test/pr-e2e-gate-fork-skip.test.ts | 23 ++- test/pr-e2e-gate-lifecycle.test.ts | 5 +- test/pr-e2e-gate-typed-target.test.ts | 95 ++++++++++ test/pr-e2e-gate-workflow.test.ts | 24 ++- test/pr-e2e-gate.test.ts | 45 ++++- test/pr-risk-plan.test.ts | 34 +++- tools/advisors/e2e-recommendations.mts | 29 ++- tools/advisors/risk-plan.mts | 75 +++++++- tools/e2e/operations-workflow-boundary.mts | 3 +- tools/e2e/pr-e2e-gate.mts | 170 +++++++++++++----- ...upload-e2e-artifacts-workflow-boundary.mts | 1 + tools/e2e/workflow-boundary.mts | 27 +-- tools/e2e/workflow-plan.mts | 16 +- tools/pr-review-advisor/analyze.mts | 36 +++- 19 files changed, 565 insertions(+), 167 deletions(-) create mode 100644 test/pr-e2e-gate-typed-target.test.ts diff --git a/.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md b/.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md index d0fd075cc90..565859e9725 100644 --- a/.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md +++ b/.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md @@ -34,9 +34,9 @@ The script handles the deterministic checks. You handle judgment calls: - **Controller workflow versus PR verdict:** `E2E / PR Gate Controller` reports whether the trusted controller safely published an outcome. It can be green while the custom `E2E / PR Gate` check is failed or cancelled. Merge authority comes from the custom check for the current head and base, not from the controller workflow. `Superseded by PR update` and `PR closed — gate no longer applies` are cancelled outcomes for an obsolete revision and require no action on that old check; the current revision still needs its own successful gate. The closed-PR outcome also covers a deleted fork repository whose GitHub head-repository field is empty. - **CI or selected E2E did not pass:** A handled prerequisite-CI or selected-E2E failure leaves `E2E / PR Gate` red while `E2E / PR Gate Controller` completes successfully. Follow the CI run, E2E run, non-passing job, and failed-step links in the custom check, fix or rerun the failing work, and require a successful gate for the resulting revision. A green controller run does not override the red verdict. - **Evidence could not be verified:** If a selected child succeeds but `Download evidence` fails, is cancelled, or is skipped, the custom check fails closed with `Evidence could not be verified` and `E2E / PR Gate Controller` is red. Inspect the `Download evidence` step and rerun the gate; the green child alone is not accepted evidence. A successful download with missing, duplicate, skipped, pending, or reported-failing signals is a handled red PR verdict with a green controller. Malformed or unsafe evidence, schema or identity mismatches, and traversal-limit violations remain controller verification errors: both the check and controller fail closed. -- **Fork PR credentialed E2E skip approval:** The primary fork path uses the same failed `E2E / PR Gate` check. Follow the explicit `E2E / PR Gate Controller run ` link embedded in the check summary rather than relying on the custom check's **Details** destination, choose **Review deployments**, select `approve-credentialed-e2e-skip-for-fork-pr`, add a comment if useful, and approve. This records that the selected credential-bearing E2E jobs will not run; it does not authorize fork code to run with repository secrets. The waiting job has `deployment: false`, no secrets, and no PR-controlled execution. The controller reads GitHub's approval history, requires one approval for only that environment, verifies that the recorded reviewer still has `maintain` or `admin`, and then applies the existing PR/head/base, deterministic-plan, matching-failed-check, compatible-`main`, and final stale-revision checks. A recorded approval completes the custom check successfully with `Credentialed E2E skipped for fork PR — approved by @` and leads with `Outcome: APPROVED SKIP — credentialed E2E did not run.` Treat this as an audited skip, not passing E2E evidence. Configure the environment before rollout with required reviewers whose approving members have `maintain` or `admin`; add no secrets, variables, or custom protection app, and preferably disable administrator bypass. If **Review deployments** is absent, the environment may be missing or unprotected, or the run may no longer be waiting. Configure it and trigger fresh upstream PR CI for a new gate run, or use the manual fork-skip fallback below. Do not rerun the waiting workflow: environment approvals are not attempt-bound, so the controller accepts only the first attempt. Per-PR concurrency cancels an older waiting approval when a newer revision reaches the gate. -- **Manual fork PR credentialed E2E skip:** Use this only when the gate has failed with `Maintainer approval required to skip credentialed E2E`. Independently review the fork change and non-secret PR CI, then prefer the protected-environment approval above. The controller rejects a closed PR, a changed head or base, an internal PR, an empty E2E plan, a non-failed gate, a controller commit that is neither current `main` nor a safely validated ancestor of it, or a recorded reviewer below maintainer role. A safe `main` advance must preserve the controller commit as its merge base, contain fewer than 300 fully enumerated changed files, and avoid `e2e-control-plane` changes; other advances fail closed. It re-reads the open PR's head and base immediately before writing success. The result records the reviewer, optional bounded comment, validated approval-run URL, plan, and jobs that did not run. For the fallback, choose **Run workflow** on `main`, select `approve-fork-e2e-skip`, and provide the fork PR number, its current 40-character head SHA as `expected_head_sha`, its current 40-character base SHA as `expected_base_sha`, and a specific 10–500-character reason. Re-read both PR SHAs immediately before dispatch. Optionally set `evidence_url` to a reviewed URL matching `https://github.com/NVIDIA/NemoClaw/actions/runs/`, or leave it blank when no supporting run exists. PR, issue, comment, job, and external URLs are rejected. The controller validates the optional URL's repository and run-ID shape but does not inspect that run's head, jobs, or conclusion. -- **E2E control-plane authorization:** The deterministic `e2e-control-plane` family remains a conservative path boundary covering E2E and PR-CI workflows, risk policy, dependency and test configuration, preparation and upload actions, and non-documentation files under `tools/e2e/` and `test/e2e/`, including shell and Python support files. Internal PRs whose matched control-plane files are drawn only from `.github/workflows/pr-e2e-gate.yaml` and `tools/e2e/pr-e2e-gate.mts` automatically dispatch the selected jobs for that SHA. Any other or mixed internal control-plane diff fails with `Maintainer authorization required to run E2E` without running selected jobs or exposing repository secrets. Independently review the revision and its non-secret CI, then choose **Run workflow** on `main`, select `run-control-plane`, and provide the PR number, current 40-character `expected_head_sha`, current 40-character `expected_base_sha`, and a specific 10–500-character `review_reason`. Re-read both SHAs immediately before dispatch. The first-attempt workflow requires the triggering actor to have `maintain` or `admin`, and rejects forks, stale or closed PRs, plans that do not require authorization, empty selections, a missing or mismatched failed gate, and an incompatible trusted controller commit. It re-reads the current head and base before dispatch. Authorization returns the existing check to in progress and runs the selected jobs through the normal wait, evidence-download, and finish path; it cannot directly record success. Only verified evidence for that SHA can make `E2E / PR Gate` green. +- **Fork PR credentialed E2E skip approval:** The primary fork path uses the same failed `E2E / PR Gate` check. Follow the explicit `E2E / PR Gate Controller run ` link embedded in the check summary rather than relying on the custom check's **Details** destination, choose **Review deployments**, select `approve-credentialed-e2e-skip-for-fork-pr`, add a comment if useful, and approve. This records that the selected credential-bearing E2E jobs and typed targets will not run; it does not authorize fork code to run with repository secrets. The waiting job has `deployment: false`, no secrets, and no PR-controlled execution. The controller reads GitHub's approval history, requires one approval for only that environment, verifies that the recorded reviewer still has `maintain` or `admin`, and then applies the existing PR/head/base, deterministic-plan, matching-failed-check, compatible-`main`, and final stale-revision checks. A recorded approval completes the custom check successfully with `Credentialed E2E skipped for fork PR — approved by @` and leads with `Outcome: APPROVED SKIP — credentialed E2E did not run.` Treat this as an audited skip, not passing E2E evidence. Configure the environment before rollout with required reviewers whose approving members have `maintain` or `admin`; add no secrets, variables, or custom protection app, and preferably disable administrator bypass. If **Review deployments** is absent, the environment may be missing or unprotected, or the run may no longer be waiting. Configure it and trigger fresh upstream PR CI for a new gate run, or use the manual fork-skip fallback below. Do not rerun the waiting workflow: environment approvals are not attempt-bound, so the controller accepts only the first attempt. Per-PR concurrency cancels an older waiting approval when a newer revision reaches the gate. +- **Manual fork PR credentialed E2E skip:** Use this only when the gate has failed with `Maintainer approval required to skip credentialed E2E`. Independently review the fork change and non-secret PR CI, then prefer the protected-environment approval above. The controller rejects a closed PR, a changed head or base, an internal PR, an empty E2E plan, a non-failed gate, a controller commit that is neither current `main` nor a safely validated ancestor of it, or a recorded reviewer below maintainer role. A safe `main` advance must preserve the controller commit as its merge base, contain fewer than 300 fully enumerated changed files, and avoid `e2e-control-plane` changes; other advances fail closed. It re-reads the open PR's head and base immediately before writing success. The result records the reviewer, optional bounded comment, validated approval-run URL, plan, and jobs and targets that did not run. For the fallback, choose **Run workflow** on `main`, select `approve-fork-e2e-skip`, and provide the fork PR number, its current 40-character head SHA as `expected_head_sha`, its current 40-character base SHA as `expected_base_sha`, and a specific 10–500-character reason. Re-read both PR SHAs immediately before dispatch. Optionally set `evidence_url` to a reviewed URL matching `https://github.com/NVIDIA/NemoClaw/actions/runs/`, or leave it blank when no supporting run exists. PR, issue, comment, job, and external URLs are rejected. The controller validates the optional URL's repository and run-ID shape but does not inspect that run's head, selected checks, or conclusion. +- **E2E control-plane authorization:** The deterministic `e2e-control-plane` family remains a conservative path boundary covering E2E and PR-CI workflows, risk policy, dependency and test configuration, preparation and upload actions, and non-documentation files under `tools/e2e/` and `test/e2e/`, including shell and Python support files. Internal PRs whose matched control-plane files are drawn only from `.github/workflows/pr-e2e-gate.yaml` and `tools/e2e/pr-e2e-gate.mts` automatically dispatch the selected jobs for that SHA. The risk plan can also select an exact allowlisted typed target for a workflow-wired check and dispatch jobs and targets together in one child run. Any other or mixed internal control-plane diff fails with `Maintainer authorization required to run E2E` without running selected checks or exposing repository secrets. Independently review the revision and its non-secret CI, then choose **Run workflow** on `main`, select `run-control-plane`, and provide the PR number, current 40-character `expected_head_sha`, current 40-character `expected_base_sha`, and a specific 10–500-character `review_reason`. Re-read both SHAs immediately before dispatch. The first-attempt workflow requires the triggering actor to have `maintain` or `admin`, and rejects forks, stale or closed PRs, plans that do not require authorization, empty selections, a missing or mismatched failed gate, and an incompatible trusted controller commit. It re-reads the current head and base before dispatch. Authorization returns the existing check to in progress and runs the selected jobs and targets through the normal wait, evidence-download, and finish path; it cannot directly record success. Only verified evidence for that SHA can make `E2E / PR Gate` green. - **Two-phase ruleset rollout and backfill:** First deploy the E2E check producer and its trusted fork handling. Re-run `CI / Pull Request` for every already-open PR head/base pair (approving a first-time fork run when necessary), then verify that `E2E / PR Gate` is attached to that same head SHA for the current base. The fail-closed maintainer checker identifies any head still needing backfill. Adding the workflow does not retroactively report the context, and enabling the ruleset first leaves existing PRs at "Waiting for status to be reported." Do not activate the context as a security boundary while it is scoped only to the shared GitHub Actions app: that app identity cannot distinguish this workflow from another workflow. First give the gate distinct provenance through a dedicated GitHub App or an organization required-workflow rule. When the context is finally activated, also enable strict/up-to-date required status checks; otherwise a successful head can remain mergeable after `main` advances and changes the effective merge diff. Preserve the control-plane review and deterministic floor even after those external protections are in place. - **Contributor compliance failed:** Reject the PR and ask the contributor to provide the PR-body DCO declaration or replace unverified commits with a clean verified history. Do not approve, merge, amend, sign, or force-push on the contributor's behalf. - **Contributor/approver overlap:** Surface `advisories.contributorApprovalOverlap` when the same account not recognized as automated by the supported login conventions appears as the current PR opener, commit author, or co-author and its latest opinionated review is approved. The invalid state detected here is contributor and approver identity overlap in the current GitHub PR metadata; the source boundary is the current opener plus all commit-author and review pages fetched through GitHub's GraphQL API. The advisory includes contributors whose commits remain in the current PR head at check time; it does not retain original push actors or authors removed when history is rebased, squashed, or fixed up. A clear result is not proof of independent approval. Missing, invalid, or conflicting review timestamps, or failure to retrieve complete paginated history, produce a warning because the latest opinion cannot be selected reliably. diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml index 110992ecd2c..dac06d6feae 100644 --- a/.github/workflows/e2e.yaml +++ b/.github/workflows/e2e.yaml @@ -123,7 +123,8 @@ jobs: [[ "$PLAN_HASH" =~ ^[a-f0-9]{64}$ ]] || { echo "::error::plan_hash must be a lowercase SHA-256"; exit 1; } [[ "$CORRELATION_ID" =~ ^[a-f0-9]{8}-[a-f0-9]{4}-4[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}$ ]] || { echo "::error::correlation_id must be a lowercase UUIDv4"; exit 1; } [[ "$PR_NUMBER" =~ ^[1-9][0-9]*$ ]] || { echo "::error::pr_number must be a positive integer"; exit 1; } - [[ -n "$JOBS" && -z "$TARGETS" ]] || { echo "::error::PR E2E runs require jobs and do not accept targets"; exit 1; } + [[ -n "$JOBS" || -n "$TARGETS" ]] || { echo "::error::PR E2E runs require controller-selected jobs or targets"; exit 1; } + [[ -z "$TARGETS" || "$TARGETS" == "ubuntu-repo-cloud-langchain-deepagents-code" ]] || { echo "::error::PR E2E target is not approved by the trusted controller"; exit 1; } pull_json="$(curl --fail --silent --show-error --proto '=https' \ --header "Authorization: Bearer ${GITHUB_TOKEN}" \ @@ -152,10 +153,6 @@ jobs: mock|internal-nvidia|public-nvidia) ;; *) echo "::error::Invalid inference_mode: ${INFERENCE_MODE}" >&2; exit 1 ;; esac - if [ -n "${JOBS}" ] && [ -n "${TARGETS}" ]; then - echo "::error::Use either targets or jobs, not both." >&2 - exit 1 - fi for selector_name in JOBS TARGETS; do selector_value="${!selector_name}" if [ -n "${selector_value}" ] && [[ ! "${selector_value}" =~ ^[A-Za-z0-9_-]+(,[A-Za-z0-9_-]+)*$ ]]; then @@ -167,7 +164,8 @@ jobs: planner_args=() if [ -n "${JOBS}" ]; then planner_args+=(--jobs "${JOBS}") - elif [ -n "${TARGETS}" ]; then + fi + if [ -n "${TARGETS}" ]; then planner_args+=(--targets "${TARGETS}") fi plan="$(npx tsx tools/e2e/workflow-plan.mts "${planner_args[@]}")" @@ -212,8 +210,9 @@ jobs: fi expected_hermes_selected=false - selected_csv="${JOBS:-${TARGETS}}" - if [ -z "${selected_csv}" ] || [[ ",${selected_csv}," == *",hermes-e2e,"* ]]; then + if { [ -z "${JOBS}" ] && [ -z "${TARGETS}" ]; } \ + || [[ ",${JOBS}," == *",hermes-e2e,"* ]] \ + || [[ ",${TARGETS}," == *",hermes-e2e,"* ]]; then expected_hermes_selected=true fi hermes_selected="$(jq -r '.hermesSelected' <<< "${plan}")" @@ -248,7 +247,7 @@ jobs: live: needs: generate-matrix - if: ${{ (github.event_name != 'workflow_dispatch' || inputs.jobs == '') && needs.generate-matrix.outputs.matrix != '[]' }} + if: ${{ needs.generate-matrix.outputs.matrix != '[]' }} runs-on: ${{ matrix.runner }} timeout-minutes: 45 strategy: @@ -371,6 +370,7 @@ jobs: - name: Run live E2E tests env: + E2E_TARGET_ID: ${{ matrix.id }} NVIDIA_INFERENCE_API_KEY: ${{ secrets.NVIDIA_INFERENCE_API_KEY }} TARGET_ID: ${{ matrix.id }} run: | @@ -465,6 +465,7 @@ jobs: e2e-artifacts/live/${{ matrix.id }}/onboarding.result.json e2e-artifacts/live/${{ matrix.id }}/state-validation.result.json e2e-artifacts/live/${{ matrix.id }}/cloud-onboard-trace-timing-summary.json + e2e-artifacts/live/risk-signal.json e2e-artifacts/live/${{ matrix.id }}/actions/ e2e-artifacts/live/${{ matrix.id }}/logs/ e2e-artifacts/live/${{ matrix.id }}/shell/ diff --git a/test/e2e/README.md b/test/e2e/README.md index 4f775e1c774..7027968b4bf 100644 --- a/test/e2e/README.md +++ b/test/e2e/README.md @@ -149,14 +149,18 @@ Shared sandbox-boundary changes have a floor of `full-e2e`, `hermes-e2e`, and family is a conservative path boundary that includes non-documentation files under `tools/e2e/` and `test/e2e/`, plus the E2E and PR-CI workflows, risk policy, dependency and test configuration, and preparation and upload actions. +The Deep Agents Code headless-inference check additionally selects the exact +`ubuntu-repo-cloud-langchain-deepagents-code` typed target. That target is +hashed into the risk plan beside the control-plane floor jobs, so the +controller dispatches both selector types in one correlated workflow run. An internal revision whose matched control-plane files are drawn only from the trusted controller and observer boundaries—`.github/workflows/pr-e2e-gate.yaml`, `tools/e2e/pr-e2e-gate.mts`, and `tools/e2e/pr-e2e-required.mts`—automatically dispatches those selected jobs. Any other or mixed internal control-plane revision requires the exact-SHA maintainer authorization below before credentialed execution begins. If no job -is selected, coordination passes without an E2E run and the native required job -mirrors that success. +or target is selected, coordination passes without an E2E run and the native +required job mirrors that success. Before dispatch, the controller verifies that the live PR still matches the CI run's exact head and base. It uses its own workflow commit when that commit is @@ -171,8 +175,10 @@ matrix or secret-bearing jobs can run, `e2e.yaml` requires `github.workflow_sha` to match that accepted commit. Each selected job checks out `checkout_sha`. The same validation verifies that the PR remains open, belongs to `NVIDIA/NemoClaw`, and still has both the dispatched head and base -commits. The dispatch includes selected jobs and valid plan and correlation -metadata, but not `targets`. The controller uses GitHub's returned run ID for +commits. The dispatch includes selected jobs, allowlisted typed targets, and +valid plan and correlation metadata. Controller-bound targets are restricted +to the trusted allowlist; ordinary branch dispatch is not an acceptable +substitute. The controller uses GitHub's returned run ID for waiting, evidence download, and completion, then revalidates that the PR is still open with the live head, base, and exact-diff coordination identity before recording a final result. The native observer revalidates the live revision @@ -181,7 +187,7 @@ before mirroring that terminal result. An internal revision whose control-plane matches include a file outside the trusted controller and observer boundaries leaves coordination in progress with `Maintainer authorization required to run E2E`. The native required job -keeps waiting for the authorization flow. No selected job runs and no +keeps waiting for the authorization flow. No selected job or target runs and no repository secret is exposed. After reviewing the exact revision, a repository maintainer or administrator chooses **Run workflow** on `main`, selects `run-control-plane`, and supplies the PR number, current 40-character head SHA @@ -191,7 +197,8 @@ first workflow attempt and revalidates the actor's `maintain` or `admin` permission, internal repository origin, open PR, exact head and base, risk plan, matching pending coordination state, compatible trusted controller commit, and final live revision. It then updates coordination to -`Running E2E job(s)` and dispatches the selected jobs. If authorization +`Running E2E check(s)` and dispatches the selected jobs and targets in +one workflow run. If authorization fails before a child run is dispatched, the controller restores the authorization title and leaves coordination in progress so a maintainer can correct the problem and launch a fresh first-attempt authorization. After a @@ -210,9 +217,10 @@ authorize them. The normal wait, evidence download, and finish path is the only path that can record success; the authorization itself cannot make the gate green. A changed head or base requires a new authorization. -A fork revision that selects jobs completes coordination as failed while the -native required job waits for the skip-approval flow. The controller does not -dispatch the selected credential-bearing jobs or expose repository secrets. +A fork revision that selects jobs or typed targets completes coordination as +failed while the native required job waits for the skip-approval flow. The +controller does not dispatch the selected credential-bearing jobs or targets +or expose repository secrets. Non-secret PR CI remains required. The failed coordination summary embeds an explicit link to the same `E2E / PR Gate Controller` run; maintainers follow that link rather than relying on the coordination check's **Details** @@ -225,10 +233,10 @@ That controller run starts `deployment: false`, the job does not create a deployment record. A maintainer opens the linked run, chooses **Review deployments**, selects that environment, and approves it. The approval records that the selected credential-bearing -jobs will not run; it does not authorize fork code to run with repository -secrets. The comment is optional, and the workflow reads both the reviewer and -comment from GitHub's run approval history rather than accepting an actor -supplied by the job. +jobs and targets will not run; it does not authorize fork code to run with +repository secrets. The comment is optional, and the workflow reads both the +reviewer and comment from GitHub's run approval history rather than accepting +an actor supplied by the job. Before rollout, create `approve-credentialed-e2e-skip-for-fork-pr` in the repository with one or more required reviewers whose approving members have @@ -254,11 +262,12 @@ still `main` or has only a compatible safe descendant as described above. Immediately before recording success, it reads the live PR again and requires the same exact head and base. The result records the reviewer, bounded optional comment, validated -approval-run URL, plan hash, and jobs that did not run. The successful skip -coordination check is titled `Credentialed E2E skipped for fork PR — approved by -@` and begins with `Outcome: APPROVED SKIP — credentialed E2E did -not run.` It never claims that the selected jobs passed. The native required -job mirrors this approved-skip success. +approval-run URL, plan hash, and jobs and targets that did not run. The +successful skip coordination check is titled +`Credentialed E2E skipped for fork PR — approved by @` and begins +with `Outcome: APPROVED SKIP — credentialed E2E did not run.` It never claims +that the selected checks passed. The native required job mirrors this +approved-skip success. The manual fork skip approval on `main` remains available as a fallback. Choose `approve-fork-e2e-skip` and provide the PR number, current `expected_head_sha`, @@ -272,21 +281,23 @@ failed-check, compatible-`main`, and final stale-revision checks. Any new commit receives a different gate and requires a new decision; a base change also invalidates the decision. -The Vitest reporter writes one `risk-signal.json` for each selected job and -matrix shard. -The checked workflow boundary requires every policy-selected job to expose its -matching job identity, attach the reporter to every Vitest invocation, and -always upload its evidence artifact. +The Vitest reporter writes one `risk-signal.json` for each selected job shard +and typed target. Typed targets bind the signal identity to the exact matrix ID +and use the `default` evidence shard. The checked workflow boundary requires +every policy-selected execution path to expose its matching identity, attach +the reporter to every Vitest invocation, and always upload its evidence +artifact. Each signal binds the observed checkout SHA, expected SHA, plan hash, correlation ID, and pass, failure, skip, pending, and unhandled-error counts. The controller retains `pr-e2e-risk-plan-` for 14 days, while each -signal travels in the selected job's existing E2E artifact. +signal travels in the selected job or target's existing E2E artifact. Its private dispatch state is protected by a SHA-256 digest that is verified before downloaded evidence is classified. -When the plan selects jobs, coordination passes only when the E2E run succeeds -and every expected job shard uploads one complete passing signal with no skips -or pending tests. The native required job passes only after observing that +When the plan selects jobs or targets, coordination passes only when the E2E +run succeeds and every expected job shard and target uploads one complete +passing signal with no skips or pending tests. The native required job passes +only after observing that trusted success. For the current exact diff, every other dispatched outcome fails. A failed coordination result links the selected E2E run and up to 10 non-passing jobs, including up to three failed step names per job. If GitHub diff --git a/test/e2e/support/e2e-workflow.test.ts b/test/e2e/support/e2e-workflow.test.ts index 3f1caa48ed9..2586cd84cb5 100644 --- a/test/e2e/support/e2e-workflow.test.ts +++ b/test/e2e/support/e2e-workflow.test.ts @@ -37,6 +37,34 @@ describe("e2e workflow boundary", () => { expect(validateE2eWorkflowBoundary()).toEqual([]); }); + it("binds typed-target evidence identity and upload to the live matrix entry", () => { + const workflow = readWorkflow() as { + jobs: Record< + string, + { + env?: Record; + steps?: Array<{ + env?: Record; + name?: string; + with?: Record; + }>; + } + >; + }; + const live = workflow.jobs.live!; + const run = live.steps!.find((step) => step.name === "Run live E2E tests")!; + run.env!.E2E_TARGET_ID = "unbound-target"; + const upload = live.steps!.find((step) => step.name === "Upload E2E artifacts")!; + upload.with!.path = upload.with!.path.replace("e2e-artifacts/live/risk-signal.json\n", ""); + + expect(validateE2eWorkflow(workflow)).toEqual( + expect.arrayContaining([ + "live E2E step must bind risk-signal identity to matrix.id", + "artifact upload path must include e2e-artifacts/live/risk-signal.json", + ]), + ); + }); + it("rejects Bedrock matrix shard identity drift (#6938)", () => { const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "e2e-bedrock-shard-workflow-")); const workflowPath = path.join(tmp, "workflow.yaml"); @@ -258,9 +286,21 @@ describe("e2e workflow boundary", () => { targets: "network-policy", }), ).toMatchObject({ - valid: false, + valid: true, liveTargetsRun: false, - selectedFreeStandingJobs: [], + selectedFreeStandingJobs: ["network-policy"], + registryTargets: [], + }); + expect( + evaluateE2eWorkflowDispatchSelectors({ + jobs: "network-policy", + targets: "ubuntu-repo-cloud-langchain-deepagents-code", + }), + ).toMatchObject({ + valid: true, + liveTargetsRun: true, + selectedFreeStandingJobs: ["network-policy"], + registryTargets: ["ubuntu-repo-cloud-langchain-deepagents-code"], }); expect( evaluateE2eWorkflowDispatchSelectors({ diff --git a/test/e2e/support/workflow-plan.test.ts b/test/e2e/support/workflow-plan.test.ts index 90f51e73d4f..e113075dd57 100644 --- a/test/e2e/support/workflow-plan.test.ts +++ b/test/e2e/support/workflow-plan.test.ts @@ -81,10 +81,13 @@ describe("E2E workflow plan", () => { expect(() => buildE2eWorkflowPlan({ [kind]: value })).toThrow(`Invalid ${kind} input`); }); - it("rejects simultaneous jobs and targets", () => { - expect(() => buildE2eWorkflowPlan({ jobs: "hermes-e2e", targets: "hermes-e2e" })).toThrow( - "Use either jobs or targets, not both", - ); + it("combines free-standing jobs and typed targets in one execution plan", () => { + const registryId = firstId(buildLiveTargetMatrix(), "supported registry target"); + const plan = buildE2eWorkflowPlan({ jobs: "hermes-e2e", targets: registryId }); + + expect(plan.matrix.map((row) => row.id)).toEqual([registryId]); + expect(plan.testMatrix).toEqual([]); + expect(plan.hermesSelected).toBe(true); }); it("emits one compact JSON line with the deterministic workflow-output schema", () => { @@ -115,12 +118,12 @@ describe("E2E workflow plan", () => { it("reports CLI failures as workflow annotations", () => { const result = spawnSync( TSX, - [PLANNER_CLI, "--jobs", "hermes-e2e", "--targets", "hermes-e2e"], + [PLANNER_CLI, "--jobs", "hermes-e2e", "--targets", "definitely-unknown-e2e-target"], { cwd: REPO_ROOT, encoding: "utf8", timeout: 30_000 }, ); expect(result.status).toBe(1); expect(result.stdout).toBe(""); - expect(result.stderr).toBe("::error::Use either jobs or targets, not both\n"); + expect(result.stderr).toContain("::error::Unknown target 'definitely-unknown-e2e-target'"); }); }); diff --git a/test/pr-e2e-gate-fork-skip.test.ts b/test/pr-e2e-gate-fork-skip.test.ts index 50e3804140c..ec84f149b53 100644 --- a/test/pr-e2e-gate-fork-skip.test.ts +++ b/test/pr-e2e-gate-fork-skip.test.ts @@ -334,7 +334,7 @@ describe("PR E2E controller fork credentialed E2E skip approval safety", () => { details_url: `https://github.com/NVIDIA/NemoClaw/actions/runs/${GATE_RUN_ID}`, output: { title: "Maintainer approval required to skip credentialed E2E", - summary: expect.stringContaining("The selected jobs were not run"), + summary: expect.stringContaining("The selected jobs and targets were not run"), }, }); expect(JSON.stringify(completion?.body)).toContain("Review deployments"); @@ -404,7 +404,7 @@ describe("PR E2E controller fork credentialed E2E skip approval safety", () => { output: { title: "Maintainer authorization required to run E2E", summary: expect.stringContaining( - "No selected E2E job ran and no repository secret was exposed", + "No selected E2E job or target ran and no repository secret was exposed", ), }, }); @@ -455,7 +455,7 @@ describe("PR E2E controller fork credentialed E2E skip approval safety", () => { expect(requests.at(-1)?.body).toMatchObject({ status: "completed", conclusion: "success", - output: { title: "No E2E jobs selected" }, + output: { title: "No E2E checks selected" }, }); } finally { fs.rmSync(workDir, { recursive: true, force: true }); @@ -691,7 +691,7 @@ describe("PR E2E controller fork credentialed E2E skip approval safety", () => { ), }, }); - expect(JSON.stringify(completion?.body)).toContain("Selected jobs not run"); + expect(JSON.stringify(completion?.body)).toContain("Selected jobs and targets not run"); expect(JSON.stringify(completion?.body)).toContain( "Approval source: manual fallback; no supporting Actions run was supplied.", ); @@ -724,7 +724,13 @@ describe("PR E2E controller fork credentialed E2E skip approval safety", () => { ), githubFetchRoute( ({ url }) => url.includes("/pulls/42/files?"), - () => githubResponse([{ filename: "test/e2e/risk-signal-reporter.ts" }]), + () => + githubResponse([ + { + filename: + "test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh", + }, + ]), ), existingPrGateCheckRunsRoute({ status: "in_progress", @@ -781,7 +787,8 @@ describe("PR E2E controller fork credentialed E2E skip approval safety", () => { expect(dispatch?.body).toMatchObject({ ref: "main", inputs: { - jobs: "cloud-onboard,credential-sanitization,security-posture", + jobs: "cloud-onboard,credential-sanitization,security-posture,inference-routing,network-policy", + targets: "ubuntu-repo-cloud-langchain-deepagents-code", pr_number: "42", checkout_sha: HEAD_SHA, base_sha: BASE_SHA, @@ -798,7 +805,7 @@ describe("PR E2E controller fork credentialed E2E skip approval safety", () => { }); expect(checkUpdates[1]?.body).toMatchObject({ status: "in_progress", - output: { title: "Running 3 E2E jobs" }, + output: { title: "Running 6 E2E checks" }, }); expect( checkUpdates.some( @@ -854,7 +861,7 @@ describe("PR E2E controller fork credentialed E2E skip approval safety", () => { (request) => { const body = request.body as Record; const title = (body.output as { title?: string } | undefined)?.title; - const updateFails = title === "Running 3 E2E jobs"; + const updateFails = title === "Running 3 E2E checks"; check = updateFails ? check : { ...check, ...body }; return updateFails ? githubResponse({ message: "simulated update failure" }, 500) diff --git a/test/pr-e2e-gate-lifecycle.test.ts b/test/pr-e2e-gate-lifecycle.test.ts index 8b3f84ae600..b294e326b41 100644 --- a/test/pr-e2e-gate-lifecycle.test.ts +++ b/test/pr-e2e-gate-lifecycle.test.ts @@ -104,7 +104,7 @@ function pullRequestListItem(pull = pullRequest()): Omit = {}): E2eRiskSignal { + return { + version: 1, + jobId: DCODE_TARGET, + shardId: "default", + expectedSha: gate.commitSha, + testedSha: gate.commitSha, + planHash: gate.planHash, + correlationId: gate.correlationId, + passed: 1, + failed: 0, + skipped: 0, + pending: 0, + unhandledErrors: 0, + runReason: "passed", + ...overrides, + }; +} + +describe("PR E2E typed-target gate", () => { + it("requires complete bound evidence for a target-only state", () => { + const gate = state(); + const target = signal(gate); + const classify = (signals: E2eRiskSignal[]) => + classifyPrGateEvidence({ + workflowConclusion: "success", + expectedJobs: gate.expectedJobs, + expectedTargets: gate.expectedTargets, + expectedShards: gate.expectedShards, + signals, + }); + + expect(validatePrGateState(gate)).toEqual(gate); + expect(validateSignal(target, gate)).toEqual(target); + expect(classify([target]).conclusion).toBe("success"); + expect(classify([]).title).toBe("Evidence is missing"); + expect(classify([signal(gate, { skipped: 1 })]).title).toBe("Evidence is incomplete"); + expect(classify([target, target]).title).toBe("Duplicate evidence"); + }); + + it("rejects an unapproved target before dispatch", async () => { + await expect( + dispatchPrGate({ + repository: "NVIDIA/NemoClaw", + token: "token", + jobs: [], + targets: ["ubuntu-repo-cloud-openclaw"], + prNumber: 42, + commitSha: HEAD_SHA, + baseSha: BASE_SHA, + workflowSha: WORKFLOW_SHA, + planHash: "c".repeat(64), + correlationId: CORRELATION_ID, + }), + ).rejects.toThrow(/Controller dispatch inputs are invalid/u); + }); +}); diff --git a/test/pr-e2e-gate-workflow.test.ts b/test/pr-e2e-gate-workflow.test.ts index e53db22e564..dc78f5ff62f 100644 --- a/test/pr-e2e-gate-workflow.test.ts +++ b/test/pr-e2e-gate-workflow.test.ts @@ -228,6 +228,7 @@ function runChildValidation( currentPullSha: string, currentPullBase = BASE_SHA, currentWorkflowSha = WORKFLOW_SHA, + selectors: { jobs?: string; targets?: string } = {}, ) { const workflow = readYaml(E2E_PATH); const validation = step(workflow.jobs["generate-matrix"], "Validate controller dispatch"); @@ -273,11 +274,11 @@ esac FAKE_PR_SHA: currentPullSha, GITHUB_REPOSITORY: "NVIDIA/NemoClaw", GITHUB_TOKEN: "token", - JOBS: "onboard-repair", + JOBS: selectors.jobs ?? "onboard-repair", PATH: `${binDir}:${process.env.PATH ?? ""}`, PLAN_HASH: "b".repeat(64), PR_NUMBER: "42", - TARGETS: "", + TARGETS: selectors.targets ?? "", WORKFLOW_EVENT: "workflow_dispatch", WORKFLOW_REF: "refs/heads/main", WORKFLOW_SHA: currentWorkflowSha, @@ -655,14 +656,33 @@ describe("PR E2E gate workflow", () => { const stale = runChildValidation("c".repeat(40)); const retargeted = runChildValidation(HEAD_SHA, "d".repeat(40)); const racedWorkflow = runChildValidation(HEAD_SHA, BASE_SHA, "e".repeat(40)); + const combined = runChildValidation(HEAD_SHA, BASE_SHA, WORKFLOW_SHA, { + jobs: "cloud-onboard,credential-sanitization,security-posture", + targets: "ubuntu-repo-cloud-langchain-deepagents-code", + }); + const unapprovedTarget = runChildValidation(HEAD_SHA, BASE_SHA, WORKFLOW_SHA, { + jobs: "onboard-repair", + targets: "ubuntu-repo-cloud-openclaw", + }); + const empty = runChildValidation(HEAD_SHA, BASE_SHA, WORKFLOW_SHA, { + jobs: "", + targets: "", + }); expect(current.status).toBe(0); + expect(combined.status).toBe(0); expect(stale.status).toBe(1); expect(stale.stdout).toContain("checkout_sha must match the PR head commit"); expect(retargeted.status).toBe(1); expect(retargeted.stdout).toContain("base_sha must match the PR base commit"); expect(racedWorkflow.status).toBe(1); expect(racedWorkflow.stdout).toContain("workflow_sha must match the trusted workflow commit"); + expect(unapprovedTarget.status).toBe(1); + expect(unapprovedTarget.stdout).toContain( + "PR E2E target is not approved by the trusted controller", + ); + expect(empty.status).toBe(1); + expect(empty.stdout).toContain("PR E2E runs require controller-selected jobs or targets"); }); // source-shape-contract: security -- Always-run finalization and private-workspace cleanup must survive every coordinate failure path diff --git a/test/pr-e2e-gate.test.ts b/test/pr-e2e-gate.test.ts index 5b01fcf3e6f..cb090175aeb 100644 --- a/test/pr-e2e-gate.test.ts +++ b/test/pr-e2e-gate.test.ts @@ -7,7 +7,12 @@ import os from "node:os"; import path from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; -import { buildRiskPlan, riskPlanRequiredJobIds } from "../tools/advisors/risk-plan.mts"; +import { + buildRiskPlan, + PR_E2E_TYPED_TARGET_IDS, + riskPlanRequiredJobIds, + riskPlanRequiredTargetIds, +} from "../tools/advisors/risk-plan.mts"; import { assertCorrelatedWorkflowRun, classifyPrGateEvidence, @@ -42,6 +47,9 @@ const CI_RUN_ID = 99; const CI_RUN_ATTEMPT = 3; const GATE_RUN_ID = 77; const CORRELATION_ID = "12345678-1234-4123-8123-123456789abc"; +const DCODE_TARGET = PR_E2E_TYPED_TARGET_IDS[0]; +const DCODE_CHECK = + "test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh"; const BROAD_FILES = [ "src/lib/onboard.ts", "src/lib/actions/upgrade-sandboxes.ts", @@ -144,7 +152,7 @@ function pullRequestListItem(pull = pullRequest()): Omit { "security-posture", "token-rotation", ]); + const targetPlan = buildRiskPlan({ headSha: HEAD_SHA, changedFiles: [DCODE_CHECK] }); + expect(validateRiskPlan(targetPlan, new Set(riskPlanRequiredJobIds(targetPlan)))).toEqual( + targetPlan, + ); + expect(riskPlanRequiredTargetIds(targetPlan)).toEqual([DCODE_TARGET]); expect(validatePrGateState(gate)).toEqual(gate); expect(() => validatePrGateState({ ...gate, prNumber: 0 })).toThrow(/PR number/u); - expect(() => validatePrGateState({ ...gate, expectedShards: {} })).toThrow(/shard jobs/u); + expect(() => validatePrGateState({ ...gate, expectedShards: {} })).toThrow(/shard selections/u); + expect(() => validatePrGateState({ ...gate, expectedTargets: ["unknown-target"] })).toThrow( + /State targets/u, + ); }); it("paginates canonical pull request files and includes both names for renames", async () => { @@ -320,11 +337,14 @@ describe("PR E2E controller", () => { it("fails closed for missing, duplicate, skipped, or failing evidence", () => { const gate = state(); - const complete = gate.expectedJobs.map((job) => signal(gate, job)); + const complete = [...gate.expectedJobs, ...gate.expectedTargets].map((job) => + signal(gate, job), + ); const classify = (signals: E2eRiskSignal[], workflowConclusion: string | null = "success") => classifyPrGateEvidence({ workflowConclusion, expectedJobs: gate.expectedJobs, + expectedTargets: gate.expectedTargets, expectedShards: gate.expectedShards, signals, }); @@ -376,6 +396,9 @@ describe("PR E2E controller", () => { "hermes-inference-switch": ["hosted", "anthropic"], "openclaw-inference-switch": ["hosted", "anthropic"], }); + expect(expectedSignalShards([], ".github/workflows/e2e.yaml", [DCODE_TARGET])).toEqual({ + [DCODE_TARGET]: ["default"], + }); const broadPlan = buildRiskPlan({ headSha: HEAD_SHA, changedFiles: BROAD_FILES }); const broadShards = expectedSignalShards(riskPlanRequiredJobIds(broadPlan)); expect(Object.keys(broadShards)).toHaveLength(13); @@ -383,7 +406,7 @@ describe("PR E2E controller", () => { expect(() => expectedSignalShards(["not-a-workflow-job"])).toThrow(/does not define/u); }); - it("dispatches every selected job with the exact base and accepted workflow SHA", async () => { + it("dispatches selected jobs and the allowlisted target with exact bound metadata", async () => { const jobs = ["onboard-repair", "onboard-resume", "full-e2e", "hermes-e2e"]; const fetchMock = vi.spyOn(globalThis, "fetch").mockImplementation( createGitHubFetchRouter([ @@ -412,6 +435,7 @@ describe("PR E2E controller", () => { repository: "NVIDIA/NemoClaw", token: "token", jobs, + targets: [DCODE_TARGET], prNumber: 42, commitSha: HEAD_SHA, baseSha: BASE_SHA, @@ -427,6 +451,7 @@ describe("PR E2E controller", () => { ref: "main", inputs: { jobs: jobs.join(","), + targets: DCODE_TARGET, pr_number: "42", checkout_sha: HEAD_SHA, base_sha: BASE_SHA, @@ -1318,7 +1343,7 @@ describe("PR E2E controller", () => { const command = startCommand(workDir); await startPrGate(command); gate = validatePrGateState(JSON.parse(fs.readFileSync(command.statePath, "utf8"))); - for (const job of gate.expectedJobs) { + for (const job of [...gate.expectedJobs, ...gate.expectedTargets]) { for (const shard of gate.expectedShards[job]!) { const directory = path.join(command.evidencePath, `${job}-${shard}`); fs.mkdirSync(directory, { recursive: true }); @@ -1345,6 +1370,7 @@ describe("PR E2E controller", () => { }); expect(gate.expectedJobs).toEqual(BROAD_JOBS); + expect(gate.expectedTargets).toEqual([]); expect(requests.filter((request) => request.url.includes("/pulls?"))).toHaveLength(1); // Finalization brackets evidence parsing with exact-diff reads so a PR update cannot // turn stale evidence into a current-revision result. @@ -1366,6 +1392,7 @@ describe("PR E2E controller", () => { expect(dispatch?.body).toMatchObject({ inputs: { jobs: BROAD_JOBS.join(","), + targets: "", pr_number: "42", checkout_sha: HEAD_SHA, base_sha: BASE_SHA, @@ -1387,7 +1414,7 @@ describe("PR E2E controller", () => { expect(checkUpdates[1]?.body).toMatchObject({ status: "in_progress", output: { - title: "Running 13 E2E jobs", + title: "Running 13 E2E checks", summary: expect.stringContaining("upgrade-stale-sandbox"), }, }); @@ -1395,8 +1422,8 @@ describe("PR E2E controller", () => { status: "completed", conclusion: "success", output: { - title: "All selected jobs passed", - summary: "Every expected job shard passed with no skips or pending tests.", + title: "All selected E2E checks passed", + summary: "Every expected E2E check shard passed with no skips or pending tests.", }, }); expect(fs.readFileSync(outputPath, "utf8")).toContain("finalized=true"); diff --git a/test/pr-risk-plan.test.ts b/test/pr-risk-plan.test.ts index e94050c53b4..cba3ebf4049 100644 --- a/test/pr-risk-plan.test.ts +++ b/test/pr-risk-plan.test.ts @@ -4,9 +4,11 @@ import { describe, expect, it } from "vitest"; import { buildRiskPlan, + PR_E2E_TYPED_TARGET_IDS, RISK_RULES, requiresCredentialedE2eAuthorization, riskPlanRequiredJobIds, + riskPlanRequiredTargetIds, } from "../tools/advisors/risk-plan.mts"; import { focusedE2eJobsForChangedFiles, @@ -26,7 +28,7 @@ describe("deterministic PR risk plan", () => { const second = plan("src/lib/onboard.ts", "src/lib/state/registry.ts"); expect(first).toEqual(second); - expect(first.version).toBe(3); + expect(first.version).toBe(4); expect(first.headSha).toBe(HEAD_SHA); expect(first.planHash).toMatch(/^[a-f0-9]{64}$/u); expect(first.changedFiles).toEqual(["src/lib/onboard.ts", "src/lib/state/registry.ts"]); @@ -38,6 +40,7 @@ describe("deterministic PR risk plan", () => { expect(result.tier).toBe(0); expect(result.families).toEqual([]); expect(result.requiredJobs).toEqual([]); + expect(result.requiredTargets).toEqual([]); }); it("keeps every live test behind the control-plane exception and preserves the cloud floor (#6446)", () => { @@ -84,6 +87,34 @@ describe("deterministic PR risk plan", () => { expect(result.planHash).not.toBe(withoutFocusedSelection.planHash); }); + it("hashes the Deep Agents headless check into its exact typed target", () => { + const changedFile = + "test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh"; + const result = plan(changedFile); + const adjacentCheck = plan( + "test/e2e/e2e-cloud-experimental/checks/08-deepagents-code-secret-boundary.sh", + ); + + expect(PR_E2E_TYPED_TARGET_IDS).toEqual(["ubuntu-repo-cloud-langchain-deepagents-code"]); + expect(riskPlanRequiredTargetIds(result)).toEqual(PR_E2E_TYPED_TARGET_IDS); + expect(result.requiredTargets).toEqual([ + expect.objectContaining({ + id: PR_E2E_TYPED_TARGET_IDS[0], + families: ["focused-e2e"], + matchedFiles: [changedFile], + }), + ]); + expect(result.families).toContainEqual( + expect.objectContaining({ + id: "focused-e2e", + requiredTargets: [...PR_E2E_TYPED_TARGET_IDS], + }), + ); + expect(riskPlanRequiredTargetIds(adjacentCheck)).toEqual([]); + expect(result.planHash).not.toBe(adjacentCheck.planHash); + expect(requiresCredentialedE2eAuthorization(result)).toBe(true); + }); + it("does not infer security or inference risk from unrelated path substrings", () => { const result = plan("src/lib/actions/sandbox/mcp-bridge-provider.ts", "src/lib/secretary.ts"); @@ -230,6 +261,7 @@ describe("deterministic PR risk plan", () => { expect(result.families).toEqual([]); expect(result.requiredJobs).toEqual([]); + expect(result.requiredTargets).toEqual([]); }); it("runs controller-only changes without credentialed E2E authorization", () => { diff --git a/tools/advisors/e2e-recommendations.mts b/tools/advisors/e2e-recommendations.mts index 08963940421..62bde0712f0 100644 --- a/tools/advisors/e2e-recommendations.mts +++ b/tools/advisors/e2e-recommendations.mts @@ -191,10 +191,12 @@ export function normalizeE2eCoverageResult( } function deterministicCoverageTests(changedFiles: string[], riskPlan: RiskPlan): E2eCoverageTest[] { - const tests: E2eCoverageTest[] = riskPlan.requiredJobs.map((job) => ({ - id: job.id, - reason: job.reasons.join(" "), - })); + const tests: E2eCoverageTest[] = [...riskPlan.requiredJobs, ...riskPlan.requiredTargets].map( + (selection) => ({ + id: selection.id, + reason: selection.reasons.join(" "), + }), + ); if (requiresCloudOnboardE2e(changedFiles) && !tests.some((test) => test.id === "cloud-onboard")) { tests.push({ id: "cloud-onboard", @@ -272,7 +274,7 @@ export function normalizeE2eTargetAdvisorResult( options.riskPlan ?? buildRiskPlan({ headSha: "target-normalize", changedFiles: metadata.changedFiles }); const deterministicRequired = mergeRecommendations( - deterministicRiskJobRecommendations(riskPlan, context), + deterministicRiskRecommendations(riskPlan, context), focusedJobs, ); const required = suppressFanout @@ -573,11 +575,11 @@ function deterministicFreeStandingJobRecommendations( return output.sort((left, right) => left.id.localeCompare(right.id)); } -function deterministicRiskJobRecommendations( +function deterministicRiskRecommendations( riskPlan: RiskPlan, context: E2eTargetNormalizationContext, ): E2eTargetRecommendation[] { - return riskPlan.requiredJobs + const jobs = riskPlan.requiredJobs .filter((job) => context.allowedJobIds.has(job.id)) .map((job) => ({ id: job.id, @@ -586,6 +588,19 @@ function deterministicRiskJobRecommendations( required: true, reason: job.reasons.join(" "), })); + const targets = riskPlan.requiredTargets + .filter((target) => { + const definition = getTarget(target.id); + return definition !== undefined && liveTargetSupport(definition).supported; + }) + .map((target) => ({ + id: target.id, + workflow: E2E_WORKFLOW, + selectorType: "target" as const, + required: true, + reason: target.reasons.join(" "), + })); + return [...jobs, ...targets]; } function suppressFanoutForFocusedJobs( diff --git a/tools/advisors/risk-plan.mts b/tools/advisors/risk-plan.mts index a22059edee0..bac8543fca1 100644 --- a/tools/advisors/risk-plan.mts +++ b/tools/advisors/risk-plan.mts @@ -3,7 +3,13 @@ import { createHash } from "node:crypto"; -export const RISK_PLAN_VERSION = 3 as const; +export const RISK_PLAN_VERSION = 4 as const; + +export const PR_E2E_TYPED_TARGET_IDS = ["ubuntu-repo-cloud-langchain-deepagents-code"] as const; + +const PR_E2E_TYPED_TARGET_ID_SET = new Set(PR_E2E_TYPED_TARGET_IDS); +const DEEPAGENTS_HEADLESS_INFERENCE_CHECK = + "test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh"; export type RiskTier = 0 | 1 | 2 | 3; export type RiskFamilyId = @@ -23,6 +29,11 @@ export type TrustedFocusedE2eJob = { matchedFiles: readonly string[]; }; +export type TrustedFocusedE2eTarget = { + id: string; + matchedFiles: readonly string[]; +}; + export type RiskPlanFamily = { id: RiskFamilyId; summary: string; @@ -30,6 +41,7 @@ export type RiskPlanFamily = { matchedFiles: string[]; invariants: string[]; requiredJobs: string[]; + requiredTargets: string[]; }; export type RiskPlanJob = { @@ -40,6 +52,8 @@ export type RiskPlanJob = { matchedFiles: string[]; }; +export type RiskPlanTarget = RiskPlanJob; + export type RiskPlan = { version: typeof RISK_PLAN_VERSION; headSha: string; @@ -48,9 +62,10 @@ export type RiskPlan = { tier: RiskTier; families: RiskPlanFamily[]; requiredJobs: RiskPlanJob[]; + requiredTargets: RiskPlanTarget[]; }; -type RiskRule = Omit & { +type RiskRule = Omit & { matches(file: string): boolean; }; @@ -91,12 +106,29 @@ const RISK_RELEVANT_TEST_FILES = new Set([ "test/e2e/risk-signal-reporter.ts", ]); const FOCUSED_E2E_SUMMARY = - "Changed workflow-wired E2E tests must execute through their trusted canonical jobs."; + "Changed workflow-wired E2E tests must execute through their trusted canonical jobs or typed targets."; const FOCUSED_E2E_INVARIANTS = [ - "the changed test remains wired to a selector declared by the trusted workflow", - "the canonical job executes the changed test rather than treating it as advisory coverage", + "the changed test remains wired to a job or typed-target selector declared by the trusted workflow", + "the canonical execution path runs the changed test rather than treating it as advisory coverage", ] as const; +export function isPrE2eTypedTargetId(value: string): boolean { + return PR_E2E_TYPED_TARGET_ID_SET.has(value); +} + +export function focusedPrE2eTargetsForChangedFiles( + changedFiles: readonly string[], +): TrustedFocusedE2eTarget[] { + return changedFiles.includes(DEEPAGENTS_HEADLESS_INFERENCE_CHECK) + ? [ + { + id: PR_E2E_TYPED_TARGET_IDS[0], + matchedFiles: [DEEPAGENTS_HEADLESS_INFERENCE_CHECK], + }, + ] + : []; +} + export const RISK_RULES: readonly RiskRule[] = [ { id: "lifecycle-state", @@ -332,21 +364,31 @@ export function buildRiskPlan(options: { matchedFiles, invariants: [...rule.invariants], requiredJobs: [...rule.requiredJobs], + requiredTargets: [], }, ]; }); const focusedE2eJobs = normalizeFocusedE2eJobs(options.focusedE2eJobs ?? [], changedFiles); + const focusedE2eTargets = normalizeFocusedE2eJobs( + focusedPrE2eTargetsForChangedFiles(changedFiles), + changedFiles, + ); const focusedFamilies: RiskPlanFamily[] = - focusedE2eJobs.length === 0 + focusedE2eJobs.length === 0 && focusedE2eTargets.length === 0 ? [] : [ { id: "focused-e2e", summary: FOCUSED_E2E_SUMMARY, tier: 2, - matchedFiles: stableUnique(focusedE2eJobs.flatMap((job) => job.matchedFiles)), + matchedFiles: stableUnique( + [...focusedE2eJobs, ...focusedE2eTargets].flatMap( + (selection) => selection.matchedFiles, + ), + ), invariants: [...FOCUSED_E2E_INVARIANTS], requiredJobs: focusedE2eJobs.map((job) => job.id), + requiredTargets: focusedE2eTargets.map((target) => target.id), }, ]; const families = [...staticFamilies, ...focusedFamilies]; @@ -383,9 +425,23 @@ export function buildRiskPlan(options: { jobs.set(selection.id, existing); } + const targets = new Map(); + for (const selection of focusedE2eTargets) { + targets.set(selection.id, { + id: selection.id, + tier: 2, + families: ["focused-e2e"], + reasons: [FOCUSED_E2E_SUMMARY], + matchedFiles: [...selection.matchedFiles], + }); + } + const requiredJobs = [...jobs.values()].sort( (left, right) => right.tier - left.tier || left.id.localeCompare(right.id), ); + const requiredTargets = [...targets.values()].sort( + (left, right) => right.tier - left.tier || left.id.localeCompare(right.id), + ); const tier = families.reduce( (highest, family) => Math.max(highest, family.tier) as RiskTier, 0, @@ -397,6 +453,7 @@ export function buildRiskPlan(options: { tier, families, requiredJobs, + requiredTargets, }; return { ...withoutHash, planHash: planDigest(withoutHash) }; @@ -406,6 +463,10 @@ export function riskPlanRequiredJobIds(plan: RiskPlan): string[] { return plan.requiredJobs.map((job) => job.id); } +export function riskPlanRequiredTargetIds(plan: RiskPlan): string[] { + return plan.requiredTargets.map((target) => target.id); +} + export function requiresCredentialedE2eAuthorization(plan: RiskPlan): boolean { const controlPlane = plan.families.find((family) => family.id === "e2e-control-plane"); return ( diff --git a/tools/e2e/operations-workflow-boundary.mts b/tools/e2e/operations-workflow-boundary.mts index 89c27cf0e53..fd5f0481d33 100644 --- a/tools/e2e/operations-workflow-boundary.mts +++ b/tools/e2e/operations-workflow-boundary.mts @@ -188,7 +188,8 @@ function validatePrGateDispatch(errors: string[], workflow: OperationsWorkflow): '"$WORKFLOW_SHA" == "$EXPECTED_WORKFLOW_SHA"', '"$(git rev-parse --verify HEAD)" == "$CHECKOUT_SHA"', '"$PR_NUMBER" =~ ^[1-9][0-9]*$', - '[[ -n "$JOBS" && -z "$TARGETS" ]]', + '[[ -n "$JOBS" || -n "$TARGETS" ]]', + '[[ -z "$TARGETS" || "$TARGETS" == "ubuntu-repo-cloud-langchain-deepagents-code" ]]', "https://api.github.com/repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}", "'.state'", "'.head.repo.full_name // \"\"'", diff --git a/tools/e2e/pr-e2e-gate.mts b/tools/e2e/pr-e2e-gate.mts index d72598fbbe5..cacfa9cc514 100755 --- a/tools/e2e/pr-e2e-gate.mts +++ b/tools/e2e/pr-e2e-gate.mts @@ -3,7 +3,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { spawn, type SpawnOptions } from "node:child_process"; +import { type SpawnOptions, spawn } from "node:child_process"; import { createHash, randomUUID } from "node:crypto"; import fs from "node:fs"; import path from "node:path"; @@ -15,10 +15,12 @@ import { githubApi, githubRestPaginated } from "../advisors/github.mts"; import { parseArgs } from "../advisors/io.mts"; import { buildRiskPlan, + isPrE2eTypedTargetId, RISK_PLAN_VERSION, type RiskPlan, requiresCredentialedE2eAuthorization, riskPlanRequiredJobIds, + riskPlanRequiredTargetIds, } from "../advisors/risk-plan.mts"; import { SHARED_E2E_JOB_ID } from "./credential-free-tests.mts"; import { readPrivateRegularFile, writePrivateRegularFile } from "./private-file.ts"; @@ -236,7 +238,7 @@ type WorkflowRunIdentity = { }; export type PrGateState = { - version: 2; + version: 3; commitSha: string; baseSha: string; workflowSha: string; @@ -244,6 +246,7 @@ export type PrGateState = { correlationId: string; prNumber: number; expectedJobs: string[]; + expectedTargets: string[]; expectedShards: Record; }; @@ -529,7 +532,7 @@ function readRegularJson(file: string, maxBytes = MAX_PLAN_BYTES): unknown { } export function validatePrGateState(value: unknown): PrGateState { - if (!isObjectRecord(value) || value.version !== 2) { + if (!isObjectRecord(value) || value.version !== 3) { throw new Error("State version is invalid"); } if (typeof value.commitSha !== "string" || !SHA_PATTERN.test(value.commitSha)) { @@ -552,28 +555,44 @@ export function validatePrGateState(value: unknown): PrGateState { } if ( !Array.isArray(value.expectedJobs) || - value.expectedJobs.length < 1 || !value.expectedJobs.every((job) => typeof job === "string" && JOB_PATTERN.test(job)) || new Set(value.expectedJobs).size !== value.expectedJobs.length ) { throw new Error("State jobs are invalid"); } + if ( + !Array.isArray(value.expectedTargets) || + !value.expectedTargets.every( + (target) => + typeof target === "string" && JOB_PATTERN.test(target) && isPrE2eTypedTargetId(target), + ) || + new Set(value.expectedTargets).size !== value.expectedTargets.length + ) { + throw new Error("State targets are invalid"); + } + const expectedSelections = [...value.expectedJobs, ...value.expectedTargets]; + if ( + expectedSelections.length < 1 || + new Set(expectedSelections).size !== expectedSelections.length + ) { + throw new Error("State E2E selections are invalid"); + } if (!isObjectRecord(value.expectedShards)) { throw new Error("State shards are invalid"); } const shardJobs = Object.keys(value.expectedShards).sort(); - if (JSON.stringify(shardJobs) !== JSON.stringify([...value.expectedJobs].sort())) { - throw new Error("State shard jobs do not match expected jobs"); + if (JSON.stringify(shardJobs) !== JSON.stringify([...expectedSelections].sort())) { + throw new Error("State shard selections do not match expected jobs and targets"); } - for (const job of value.expectedJobs) { - const shards = value.expectedShards[job]; + for (const selection of expectedSelections) { + const shards = value.expectedShards[selection]; if ( !Array.isArray(shards) || shards.length < 1 || new Set(shards).size !== shards.length || !shards.every((shard) => typeof shard === "string" && SHARD_PATTERN.test(shard)) ) { - throw new Error(`State shards are invalid for ${job}`); + throw new Error(`State shards are invalid for ${selection}`); } } return value as PrGateState; @@ -610,21 +629,52 @@ export function validateRiskPlan(value: unknown, allowedJobs: ReadonlySet 0 ? `jobs: ${jobs.join(", ")}` : "", + targets.length > 0 ? `targets: ${targets.join(", ")}` : "", + ] + .filter(Boolean) + .join("; "); +} + export function validateSignal( value: unknown, state: Pick< PrGateState, - "commitSha" | "planHash" | "correlationId" | "expectedJobs" | "expectedShards" + | "commitSha" + | "planHash" + | "correlationId" + | "expectedJobs" + | "expectedTargets" + | "expectedShards" >, ): E2eRiskSignal { if (!isObjectRecord(value) || value.version !== 1) { throw new Error("invalid E2E signal version"); } const signal = value as E2eRiskSignal; - if (!state.expectedJobs.includes(signal.jobId)) throw new Error("E2E signal job is unexpected"); + if (![...state.expectedJobs, ...state.expectedTargets].includes(signal.jobId)) { + throw new Error("E2E signal job or target is unexpected"); + } if (!state.expectedShards[signal.jobId]?.includes(signal.shardId)) { throw new Error("E2E signal shard is unexpected"); } @@ -648,6 +698,7 @@ export function validateSignal( export function classifyPrGateEvidence(options: { workflowConclusion: string | null; expectedJobs: readonly string[]; + expectedTargets?: readonly string[]; expectedShards: Readonly>; signals: readonly E2eRiskSignal[]; }): PrGateVerdict { @@ -658,17 +709,18 @@ export function classifyPrGateEvidence(options: { summary: `The run concluded ${options.workflowConclusion ?? "without a result"}.`, }; } - const expectedEvidence = options.expectedJobs.flatMap((job) => - (options.expectedShards[job] ?? []).map((shard) => `${job}:${shard}`), + const expectedSelections = [...options.expectedJobs, ...(options.expectedTargets ?? [])]; + const expectedEvidence = expectedSelections.flatMap((selection) => + (options.expectedShards[selection] ?? []).map((shard) => `${selection}:${shard}`), ); if ( - options.expectedJobs.length === 0 || - options.expectedJobs.some((job) => (options.expectedShards[job]?.length ?? 0) === 0) + expectedSelections.length === 0 || + expectedSelections.some((selection) => (options.expectedShards[selection]?.length ?? 0) === 0) ) { return { conclusion: "failure", title: "Evidence policy is incomplete", - summary: "At least one selected job has no configured shard policy.", + summary: "At least one selected E2E check has no configured shard policy.", }; } const byJobShard = new Map(); @@ -717,8 +769,8 @@ export function classifyPrGateEvidence(options: { } return { conclusion: "success", - title: "All selected jobs passed", - summary: "Every expected job shard passed with no skips or pending tests.", + title: "All selected E2E checks passed", + summary: "Every expected E2E check shard passed with no skips or pending tests.", }; } @@ -948,17 +1000,23 @@ async function completeCheck( async function updateRunningCheck( context: { repository: string; checkRunId: number }, token: string, - options: { childRunId: number; jobs: readonly string[]; planHash: string }, + options: { + childRunId: number; + jobs: readonly string[]; + targets: readonly string[]; + planHash: string; + }, ): Promise { const childRunUrl = `https://github.com/${context.repository}/actions/runs/${options.childRunId}`; + const selectionCount = options.jobs.length + options.targets.length; await githubApi(`repos/${context.repository}/check-runs/${context.checkRunId}`, token, { method: "PATCH", body: { status: "in_progress", details_url: childRunUrl, output: { - title: `Running ${options.jobs.length} E2E ${options.jobs.length === 1 ? "job" : "jobs"}`, - summary: `Risk plan ${options.planHash} selected: ${options.jobs.join(", ")}.`, + title: `Running ${selectionCount} E2E ${selectionCount === 1 ? "check" : "checks"}`, + summary: `Risk plan ${options.planHash} selected jobs: ${options.jobs.join(", ") || "none"}; targets: ${options.targets.join(", ") || "none"}.`, }, }, userAgent: USER_AGENT, @@ -1448,11 +1506,21 @@ function assertPullUnchanged(before: PullRequest, after: PullRequest): void { export function expectedSignalShards( jobIds: readonly string[], workflowPath = ".github/workflows/e2e.yaml", + targetIds: readonly string[] = [], ): Record { + const selections = [...jobIds, ...targetIds]; + if (new Set(selections).size !== selections.length) { + throw new Error("E2E evidence jobs and targets must be unique"); + } + for (const targetId of targetIds) { + if (!isPrE2eTypedTargetId(targetId)) { + throw new Error(`PR E2E target is not approved: ${targetId}`); + } + } const workflow = YAML.parse(fs.readFileSync(workflowPath, "utf8")) as unknown; const jobs = isObjectRecord(workflow) && isObjectRecord(workflow.jobs) ? workflow.jobs : {}; const inventory = readFreeStandingJobsInventory(workflowPath); - return Object.fromEntries( + const jobShards = Object.fromEntries( jobIds.map((jobId) => { const executionJobId = inventory.targetToJob.get(jobId) ?? jobId; if (!isObjectRecord(jobs[executionJobId])) { @@ -1513,6 +1581,10 @@ export function expectedSignalShards( return [jobId, shards]; }), ); + return { + ...jobShards, + ...Object.fromEntries(targetIds.map((targetId) => [targetId, ["default"]])), + }; } export function validateWorkflowDispatchDetails( @@ -1663,6 +1735,7 @@ export async function dispatchPrGate(options: { repository: string; token: string; jobs: readonly string[]; + targets?: readonly string[]; prNumber: number; commitSha: string; baseSha: string; @@ -1671,11 +1744,15 @@ export async function dispatchPrGate(options: { correlationId: string; }): Promise<{ runId: number; workflowSha: string }> { assertRepository(options.repository, "repository"); + const targets = options.targets ?? []; if ( !options.token || - options.jobs.length < 1 || + options.jobs.length + targets.length < 1 || new Set(options.jobs).size !== options.jobs.length || options.jobs.some((job) => !JOB_PATTERN.test(job)) || + new Set(targets).size !== targets.length || + targets.some((target) => !JOB_PATTERN.test(target) || !isPrE2eTypedTargetId(target)) || + options.jobs.some((job) => targets.includes(job)) || !Number.isSafeInteger(options.prNumber) || options.prNumber < 1 || !SHA_PATTERN.test(options.commitSha) || @@ -1700,6 +1777,7 @@ export async function dispatchPrGate(options: { ref: "main", inputs: { jobs: options.jobs.join(","), + targets: targets.join(","), pr_number: String(options.prNumber), checkout_sha: options.commitSha, base_sha: options.baseSha, @@ -1879,7 +1957,8 @@ async function dispatchSelectedPrGate(options: { paths: ControllerPaths; }): Promise { const jobs = riskPlanRequiredJobIds(options.plan); - const expectedShards = expectedSignalShards(jobs); + const targets = riskPlanRequiredTargetIds(options.plan); + const expectedShards = expectedSignalShards(jobs, E2E_WORKFLOW_PATH, targets); const correlationId = randomUUID(); if (!CORRELATION_PATTERN.test(correlationId)) { throw new Error("generated correlation ID is invalid"); @@ -1888,6 +1967,7 @@ async function dispatchSelectedPrGate(options: { repository: options.repository, token: options.token, jobs, + targets, prNumber: options.pull.number, commitSha: options.pull.head.sha, baseSha: options.baseSha, @@ -1899,7 +1979,7 @@ async function dispatchSelectedPrGate(options: { try { appendOutput("run_id", String(childRunId)); const state: PrGateState = { - version: 2, + version: 3, commitSha: options.pull.head.sha, baseSha: options.baseSha, workflowSha: dispatch.workflowSha, @@ -1907,6 +1987,7 @@ async function dispatchSelectedPrGate(options: { correlationId, prNumber: options.pull.number, expectedJobs: jobs, + expectedTargets: targets, expectedShards, }; const serializedState = `${JSON.stringify(state, null, 2)}\n`; @@ -1917,13 +1998,14 @@ async function dispatchSelectedPrGate(options: { { childRunId, jobs, + targets, planHash: options.plan.planHash, }, ); appendOutput("state_hash", sha256(serializedState)); appendOutput("dispatched", "true"); console.log( - `Run dispatched: pr=${options.pull.number} run=${childRunId} plan=${options.plan.planHash} jobs=${jobs.join(",")} url=https://github.com/${options.repository}/actions/runs/${childRunId}`, + `Run dispatched: pr=${options.pull.number} run=${childRunId} plan=${options.plan.planHash} jobs=${jobs.join(",")} targets=${targets.join(",")} url=https://github.com/${options.repository}/actions/runs/${childRunId}`, ); } catch (error) { try { @@ -2011,7 +2093,7 @@ export async function startPrGate( { repository, checkRunId }, token, "Evaluating PR commit", - "Validating the exact PR revision and selecting deterministic E2E jobs.", + "Validating the exact PR revision and selecting deterministic E2E jobs and typed targets.", ); let finalized = false; @@ -2074,6 +2156,9 @@ export async function startPrGate( ); writePrivateRegularFile(command.planPath, `${JSON.stringify(plan, null, 2)}\n`); const jobs = riskPlanRequiredJobIds(plan); + const targets = riskPlanRequiredTargetIds(plan); + const selections = riskPlanSelectionIds(plan); + const selectionSummary = riskPlanSelectionSummary(plan); const currentPull = await resolvePullRequest({ repository, token, @@ -2082,7 +2167,7 @@ export async function startPrGate( headBranch: command.headBranch, }); assertPullUnchanged(pull, currentPull); - if (command.headRepository !== repository && jobs.length > 0) { + if (command.headRepository !== repository && selections.length > 0) { const gateRunUrl = `https://github.com/${repository}/actions/runs/${command.gateRunId}`; const gateRunLink = `[${WORKFLOW_NAME} run ${command.gateRunId}](${gateRunUrl})`; await completeCheck( @@ -2092,8 +2177,8 @@ export async function startPrGate( conclusion: "failure", title: "Maintainer approval required to skip credentialed E2E", summary: [ - `This fork PR diff (head ${command.headSha}, base ${ciIdentity.baseSha}) selected credential-bearing E2E jobs: ${jobs.join(", ")}.`, - "The selected jobs were not run. No fork code received repository secrets.", + `This fork PR diff (head ${command.headSha}, base ${ciIdentity.baseSha}) selected credential-bearing E2E checks (${selectionSummary}).`, + "The selected jobs and targets were not run. No fork code received repository secrets.", `Open ${gateRunLink}, choose Review deployments, and approve the \`${PR_GATE_APPROVAL_ENVIRONMENT}\` environment to record this skip. If Review deployments is absent, the environment is unprotected or the run is no longer waiting; configure it and trigger fresh PR CI. GitHub records the reviewer and optional comment. The manual \`approve-fork-e2e-skip\` workflow operation remains available as fallback.`, ].join("\n\n"), }, @@ -2104,7 +2189,7 @@ export async function startPrGate( appendOutput("finalized", "true"); finalized = true; console.log( - `Fork not dispatched: pr=${pull.number} sha=${command.headSha} plan=${plan.planHash} jobs=${jobs.join(",")}`, + `Fork not dispatched: pr=${pull.number} sha=${command.headSha} plan=${plan.planHash} jobs=${jobs.join(",")} targets=${targets.join(",")}`, ); return; } @@ -2116,9 +2201,9 @@ export async function startPrGate( token, CONTROL_PLANE_AUTHORIZATION_TITLE, [ - `This exact internal diff (head \`${command.headSha}\`, base \`${ciIdentity.baseSha}\`) changes code that the selected credential-bearing E2E jobs execute or trust: ${jobs.join(", ")}.`, - "No selected E2E job ran and no repository secret was exposed.", - `A repository maintainer or administrator must review this exact revision, then open the [${WORKFLOW_NAME}](${workflowUrl}) workflow and run \`run-control-plane\` with the PR number, exact head and base SHAs, and a review reason. That authorized run dispatches the selected jobs and this gate passes only if their exact-SHA evidence verifies successfully.`, + `This exact internal diff (head \`${command.headSha}\`, base \`${ciIdentity.baseSha}\`) changes code that the selected credential-bearing E2E jobs or targets execute or trust (${selectionSummary}).`, + "No selected E2E job or target ran and no repository secret was exposed.", + `A repository maintainer or administrator must review this exact revision, then open the [${WORKFLOW_NAME}](${workflowUrl}) workflow and run \`run-control-plane\` with the PR number, exact head and base SHAs, and a review reason. That authorized run dispatches the selected jobs and targets in one bound workflow run, and this gate passes only if their exact-SHA evidence verifies successfully.`, `Deterministic plan: \`${plan.planHash}\`.`, ].join("\n\n"), ); @@ -2126,14 +2211,14 @@ export async function startPrGate( appendOutput("finalized", "true"); finalized = true; console.log( - `Control-plane authorization required: pr=${pull.number} sha=${command.headSha} plan=${plan.planHash} jobs=${jobs.join(",")}`, + `Control-plane authorization required: pr=${pull.number} sha=${command.headSha} plan=${plan.planHash} jobs=${jobs.join(",")} targets=${targets.join(",")}`, ); return; } - if (jobs.length === 0) { + if (selections.length === 0) { await completeCheck({ repository, checkRunId }, token, { conclusion: "success", - title: "No E2E jobs selected", + title: "No E2E checks selected", summary: "No changed files matched an E2E risk rule.", }); appendOutput("dispatched", "false"); @@ -2213,8 +2298,9 @@ export async function startControlPlanePrGate(command: ControlPlaneDispatchComma throw new Error("pull request does not require credentialed E2E authorization"); } const jobs = riskPlanRequiredJobIds(plan); - if (jobs.length === 0) { - throw new Error("authorized control-plane plan selected no E2E jobs"); + const targets = riskPlanRequiredTargetIds(plan); + if (jobs.length + targets.length === 0) { + throw new Error("authorized control-plane plan selected no E2E jobs or targets"); } writePrivateRegularFile(command.planPath, `${JSON.stringify(plan, null, 2)}\n`); const currentPull = await requireLiveExactDiff({ @@ -2450,6 +2536,7 @@ export async function finishPrGate(options: { verdict = classifyPrGateEvidence({ workflowConclusion, expectedJobs: state.expectedJobs, + expectedTargets: state.expectedTargets, expectedShards: state.expectedShards, signals, }); @@ -2689,7 +2776,8 @@ async function completeForkE2ESkip(command: ForkSkipCommand): Promise { allowedJobs, ); const jobs = riskPlanRequiredJobIds(plan); - if (jobs.length === 0) { + const targets = riskPlanRequiredTargetIds(plan); + if (jobs.length + targets.length === 0) { throw new Error("pull request does not require a credentialed E2E skip"); } const currentPull = validatePullRequest( @@ -2726,7 +2814,7 @@ async function completeForkE2ESkip(command: ForkSkipCommand): Promise { : "Approval source: manual fallback; no supporting Actions run was supplied."; const title = `Credentialed E2E skipped for fork PR — approved by @${command.maintainer}`; const approval = `Maintainer @${command.maintainer} approved skipping credentialed E2E for fork head \`${command.headSha}\` on base \`${command.baseSha}\`.`; - const nonExecution = `Selected jobs not run: ${jobs.join(", ")}.`; + const nonExecution = `Selected jobs and targets not run: ${riskPlanSelectionSummary(plan)}.`; await compatibleMainWorkflowCommit(repository, token, command.workflowSha); const finalPull = await requireLiveExactDiff({ repository, diff --git a/tools/e2e/upload-e2e-artifacts-workflow-boundary.mts b/tools/e2e/upload-e2e-artifacts-workflow-boundary.mts index cfa592b9d0f..c2c7a129aa1 100644 --- a/tools/e2e/upload-e2e-artifacts-workflow-boundary.mts +++ b/tools/e2e/upload-e2e-artifacts-workflow-boundary.mts @@ -66,6 +66,7 @@ const EXPLICIT_UPLOAD_CONTRACTS = new Map([ "e2e-artifacts/live/${{ matrix.id }}/onboarding.result.json", "e2e-artifacts/live/${{ matrix.id }}/state-validation.result.json", "e2e-artifacts/live/${{ matrix.id }}/cloud-onboard-trace-timing-summary.json", + "e2e-artifacts/live/risk-signal.json", "e2e-artifacts/live/${{ matrix.id }}/actions/", "e2e-artifacts/live/${{ matrix.id }}/logs/", "e2e-artifacts/live/${{ matrix.id }}/shell/", diff --git a/tools/e2e/workflow-boundary.mts b/tools/e2e/workflow-boundary.mts index 92d1b6875a9..a83ff9706e8 100644 --- a/tools/e2e/workflow-boundary.mts +++ b/tools/e2e/workflow-boundary.mts @@ -354,9 +354,6 @@ export function evaluateE2eWorkflowDispatchSelectors(input: { const targets = input.targets ?? ""; const errors: string[] = []; - if (jobs && targets) { - errors.push("Use either targets or jobs, not both"); - } if (targets && !SELECTOR_PATTERN.test(targets)) { errors.push("Invalid target input"); } @@ -393,17 +390,7 @@ export function evaluateE2eWorkflowDispatchSelectors(input: { }; } - if (jobs) { - return { - valid: true, - errors: [], - selectedFreeStandingJobs: splitSelector(jobs).sort(), - registryTargets: [], - liveTargetsRun: false, - }; - } - - const selectedFreeStandingJobs = new Set(); + const selectedFreeStandingJobs = new Set(splitSelector(jobs)); const registryTargets: string[] = []; for (const target of splitSelector(targets)) { const job = freeStandingTargetToJob.get(target); @@ -3816,7 +3803,6 @@ export function validateE2eWorkflow(workflowValue: unknown): string[] { } validateInferenceModeGeneration(errors, generate, generateEnv); requireRunContains(errors, generate, "npx tsx tools/e2e/workflow-plan.mts"); - requireRunContains(errors, generate, "Use either targets or jobs, not both"); requireRunContains(errors, generate, "for selector_name in JOBS TARGETS"); requireRunContains(errors, generate, "Invalid ${selector_name,,} input; use comma-separated ids"); requireRunContains(errors, generate, 'planner_args+=(--jobs "${JOBS}")'); @@ -3860,11 +3846,8 @@ export function validateE2eWorkflow(workflowValue: unknown): string[] { if (liveTargets.needs !== "generate-matrix") { errors.push("live job must depend on generate-matrix"); } - if ( - liveTargets.if !== - "${{ (github.event_name != 'workflow_dispatch' || inputs.jobs == '') && needs.generate-matrix.outputs.matrix != '[]' }}" - ) { - errors.push("live job must not run when a free-standing jobs selector is supplied"); + if (liveTargets.if !== "${{ needs.generate-matrix.outputs.matrix != '[]' }}") { + errors.push("live job must run whenever the trusted planner emits typed targets"); } const strategy = asRecord(liveTargets.strategy); if (strategy["fail-fast"] !== false) { @@ -4015,6 +3998,9 @@ export function validateE2eWorkflow(workflowValue: unknown): string[] { errors.push("live DCode profile import gate must run before live E2E tests"); } const runVitestEnv = asRecord(runVitest?.env); + if (runVitestEnv.E2E_TARGET_ID !== "${{ matrix.id }}") { + errors.push("live E2E step must bind risk-signal identity to matrix.id"); + } if (runVitestEnv.TARGET_ID !== "${{ matrix.id }}") { errors.push("live E2E step must pass matrix.id through TARGET_ID env"); } @@ -4144,6 +4130,7 @@ export function validateE2eWorkflow(workflowValue: unknown): string[] { uploadPath, "e2e-artifacts/live/${{ matrix.id }}/cloud-onboard-trace-timing-summary.json", ); + requireUploadPathContains(errors, uploadPath, "e2e-artifacts/live/risk-signal.json"); requireUploadPathContains(errors, uploadPath, "e2e-artifacts/live/${{ matrix.id }}/actions/"); requireUploadPathContains(errors, uploadPath, "e2e-artifacts/live/${{ matrix.id }}/logs/"); requireUploadPathContains(errors, uploadPath, "e2e-artifacts/live/${{ matrix.id }}/shell/"); diff --git a/tools/e2e/workflow-plan.mts b/tools/e2e/workflow-plan.mts index 50fdc120782..c6333c34c6b 100644 --- a/tools/e2e/workflow-plan.mts +++ b/tools/e2e/workflow-plan.mts @@ -48,9 +48,6 @@ function selectTestRows( export function buildE2eWorkflowPlan(selectors: WorkflowPlanSelectors = {}): E2eWorkflowPlan { const jobs = selectorIds(selectors.jobs, "jobs"); const targets = selectorIds(selectors.targets, "targets"); - if (jobs.length > 0 && targets.length > 0) { - throw new Error("Use either jobs or targets, not both"); - } const inventory = readFreeStandingJobsInventory(); const credentialFreeTests = discoverCredentialFreeTests(); @@ -64,21 +61,14 @@ export function buildE2eWorkflowPlan(selectors: WorkflowPlanSelectors = {}): E2e ); } } - - return { - matrix: [], - testMatrix: selectTestRows(credentialFreeTests, jobs), - hermesSelected: jobs.includes(HERMES_JOB_ID), - explicitOnlyJobs: [...inventory.explicitOnlyJobs], - }; } - if (targets.length > 0) { + if (jobs.length > 0 || targets.length > 0) { const registryTargets = targets.filter((target) => !inventory.targetToJob.has(target)); return { matrix: registryTargets.length > 0 ? buildLiveTargetMatrix(registryTargets) : [], - testMatrix: selectTestRows(credentialFreeTests, targets), - hermesSelected: targets.includes(HERMES_JOB_ID), + testMatrix: selectTestRows(credentialFreeTests, [...jobs, ...targets]), + hermesSelected: [...jobs, ...targets].includes(HERMES_JOB_ID), explicitOnlyJobs: [...inventory.explicitOnlyJobs], }; } diff --git a/tools/pr-review-advisor/analyze.mts b/tools/pr-review-advisor/analyze.mts index a511ff72e11..9ea10a2f2b3 100755 --- a/tools/pr-review-advisor/analyze.mts +++ b/tools/pr-review-advisor/analyze.mts @@ -7,8 +7,8 @@ import path from "node:path"; import { fileURLToPath, pathToFileURL } from "node:url"; import { E2E_RENDER_LIMIT, - type E2eCoverageResult, type E2eChangedCredentialFreeTest, + type E2eCoverageResult, type E2eTargetAdvisorResult, normalizeE2eCoverageResult, normalizeE2eTargetAdvisorResult, @@ -901,19 +901,28 @@ export function classifyTestDepth( suggestedTests: ["Run the relevant existing unit/doc validation for the touched files."], }; } - if (riskPlan.requiredJobs.length > 0) { + if (riskPlan.requiredJobs.length > 0 || riskPlan.requiredTargets.length > 0) { return { verdict: "runtime_validation_recommended", rationale: `Deterministic regression risks require live validation: ${riskPlan.families .map((family) => family.id) .join(", ")}.`, - suggestedTests: riskPlan.requiredJobs.map( - (job) => - `Run the \`${job.id}\` E2E job for ${job.reasons.join("; ")} Matched files: ${job.matchedFiles - .slice(0, 5) - .map((file) => `\`${file}\``) - .join(", ")}.`, - ), + suggestedTests: [ + ...riskPlan.requiredJobs.map( + (job) => + `Run the \`${job.id}\` E2E job for ${job.reasons.join("; ")} Matched files: ${job.matchedFiles + .slice(0, 5) + .map((file) => `\`${file}\``) + .join(", ")}.`, + ), + ...riskPlan.requiredTargets.map( + (target) => + `Run the \`${target.id}\` typed E2E target for ${target.reasons.join("; ")} Matched files: ${target.matchedFiles + .slice(0, 5) + .map((file) => `\`${file}\``) + .join(", ")}.`, + ), + ], }; } const e2eSignals = sourceFiles.filter( @@ -2109,6 +2118,7 @@ function buildReconciliationTurnContext( tier: context.riskPlan.tier, familyIds: context.riskPlan.families.map((family) => family.id), requiredJobIds: context.riskPlan.requiredJobs.map((job) => job.id), + requiredTargetIds: context.riskPlan.requiredTargets.map((target) => target.id), }, linkedIssues: (context.github?.linkedIssues ?? []).map(({ number, fetchError }) => ({ number, @@ -2132,6 +2142,7 @@ export function buildRiskPlanReviewContext(plan: RiskPlan): Record ({ id: job.id, @@ -2140,6 +2151,13 @@ export function buildRiskPlanReviewContext(plan: RiskPlan): Record ({ + id: target.id, + tier: target.tier, + families: target.families, + reasons: target.reasons, + matchedFileCount: target.matchedFiles.length, + })), }; } From 7e2251e64757a3337880a1c1882ad4c5b21cccce Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Thu, 16 Jul 2026 13:51:11 -0700 Subject: [PATCH 2/7] fix(ci): bind typed target matrix to controller Signed-off-by: Carlos Villela --- .../nemoclaw-maintainer-day/MERGE-GATE.md | 2 +- .github/workflows/e2e.yaml | 9 ++++ test/e2e/README.md | 6 ++- ...e2e-report-to-pr-workflow-boundary.test.ts | 50 +++++++++++++++++-- test/e2e/support/e2e-workflow.test.ts | 24 +++++++++ test/pr-e2e-gate-typed-target.test.ts | 2 +- test/pr-e2e-gate.test.ts | 2 +- tools/e2e/workflow-boundary.mts | 11 ++++ 8 files changed, 97 insertions(+), 9 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md b/.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md index 565859e9725..6bf3fad9276 100644 --- a/.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md +++ b/.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md @@ -36,7 +36,7 @@ The script handles the deterministic checks. You handle judgment calls: - **Evidence could not be verified:** If a selected child succeeds but `Download evidence` fails, is cancelled, or is skipped, the custom check fails closed with `Evidence could not be verified` and `E2E / PR Gate Controller` is red. Inspect the `Download evidence` step and rerun the gate; the green child alone is not accepted evidence. A successful download with missing, duplicate, skipped, pending, or reported-failing signals is a handled red PR verdict with a green controller. Malformed or unsafe evidence, schema or identity mismatches, and traversal-limit violations remain controller verification errors: both the check and controller fail closed. - **Fork PR credentialed E2E skip approval:** The primary fork path uses the same failed `E2E / PR Gate` check. Follow the explicit `E2E / PR Gate Controller run ` link embedded in the check summary rather than relying on the custom check's **Details** destination, choose **Review deployments**, select `approve-credentialed-e2e-skip-for-fork-pr`, add a comment if useful, and approve. This records that the selected credential-bearing E2E jobs and typed targets will not run; it does not authorize fork code to run with repository secrets. The waiting job has `deployment: false`, no secrets, and no PR-controlled execution. The controller reads GitHub's approval history, requires one approval for only that environment, verifies that the recorded reviewer still has `maintain` or `admin`, and then applies the existing PR/head/base, deterministic-plan, matching-failed-check, compatible-`main`, and final stale-revision checks. A recorded approval completes the custom check successfully with `Credentialed E2E skipped for fork PR — approved by @` and leads with `Outcome: APPROVED SKIP — credentialed E2E did not run.` Treat this as an audited skip, not passing E2E evidence. Configure the environment before rollout with required reviewers whose approving members have `maintain` or `admin`; add no secrets, variables, or custom protection app, and preferably disable administrator bypass. If **Review deployments** is absent, the environment may be missing or unprotected, or the run may no longer be waiting. Configure it and trigger fresh upstream PR CI for a new gate run, or use the manual fork-skip fallback below. Do not rerun the waiting workflow: environment approvals are not attempt-bound, so the controller accepts only the first attempt. Per-PR concurrency cancels an older waiting approval when a newer revision reaches the gate. - **Manual fork PR credentialed E2E skip:** Use this only when the gate has failed with `Maintainer approval required to skip credentialed E2E`. Independently review the fork change and non-secret PR CI, then prefer the protected-environment approval above. The controller rejects a closed PR, a changed head or base, an internal PR, an empty E2E plan, a non-failed gate, a controller commit that is neither current `main` nor a safely validated ancestor of it, or a recorded reviewer below maintainer role. A safe `main` advance must preserve the controller commit as its merge base, contain fewer than 300 fully enumerated changed files, and avoid `e2e-control-plane` changes; other advances fail closed. It re-reads the open PR's head and base immediately before writing success. The result records the reviewer, optional bounded comment, validated approval-run URL, plan, and jobs and targets that did not run. For the fallback, choose **Run workflow** on `main`, select `approve-fork-e2e-skip`, and provide the fork PR number, its current 40-character head SHA as `expected_head_sha`, its current 40-character base SHA as `expected_base_sha`, and a specific 10–500-character reason. Re-read both PR SHAs immediately before dispatch. Optionally set `evidence_url` to a reviewed URL matching `https://github.com/NVIDIA/NemoClaw/actions/runs/`, or leave it blank when no supporting run exists. PR, issue, comment, job, and external URLs are rejected. The controller validates the optional URL's repository and run-ID shape but does not inspect that run's head, selected checks, or conclusion. -- **E2E control-plane authorization:** The deterministic `e2e-control-plane` family remains a conservative path boundary covering E2E and PR-CI workflows, risk policy, dependency and test configuration, preparation and upload actions, and non-documentation files under `tools/e2e/` and `test/e2e/`, including shell and Python support files. Internal PRs whose matched control-plane files are drawn only from `.github/workflows/pr-e2e-gate.yaml` and `tools/e2e/pr-e2e-gate.mts` automatically dispatch the selected jobs for that SHA. The risk plan can also select an exact allowlisted typed target for a workflow-wired check and dispatch jobs and targets together in one child run. Any other or mixed internal control-plane diff fails with `Maintainer authorization required to run E2E` without running selected checks or exposing repository secrets. Independently review the revision and its non-secret CI, then choose **Run workflow** on `main`, select `run-control-plane`, and provide the PR number, current 40-character `expected_head_sha`, current 40-character `expected_base_sha`, and a specific 10–500-character `review_reason`. Re-read both SHAs immediately before dispatch. The first-attempt workflow requires the triggering actor to have `maintain` or `admin`, and rejects forks, stale or closed PRs, plans that do not require authorization, empty selections, a missing or mismatched failed gate, and an incompatible trusted controller commit. It re-reads the current head and base before dispatch. Authorization returns the existing check to in progress and runs the selected jobs and targets through the normal wait, evidence-download, and finish path; it cannot directly record success. Only verified evidence for that SHA can make `E2E / PR Gate` green. +- **E2E control-plane authorization:** The deterministic `e2e-control-plane` family remains a conservative path boundary covering E2E and PR-CI workflows, risk policy, dependency and test configuration, preparation and upload actions, and non-documentation files under `tools/e2e/` and `test/e2e/`, including shell and Python support files. Internal PRs whose matched control-plane files are drawn only from `.github/workflows/pr-e2e-gate.yaml`, `tools/e2e/pr-e2e-gate.mts`, and `tools/e2e/pr-e2e-required.mts` automatically dispatch the selected jobs for that SHA. The risk plan can also select an exact allowlisted typed target for a workflow-wired check and dispatch jobs and targets together in one child run. Any other or mixed internal control-plane diff fails with `Maintainer authorization required to run E2E` without running selected checks or exposing repository secrets. Independently review the revision and its non-secret CI, then choose **Run workflow** on `main`, select `run-control-plane`, and provide the PR number, current 40-character `expected_head_sha`, current 40-character `expected_base_sha`, and a specific 10–500-character `review_reason`. Re-read both SHAs immediately before dispatch. The first-attempt workflow requires the triggering actor to have `maintain` or `admin`, and rejects forks, stale or closed PRs, plans that do not require authorization, empty selections, a missing or mismatched failed gate, and an incompatible trusted controller commit. It re-reads the current head and base before dispatch. Authorization returns the existing check to in progress and runs the selected jobs and targets through the normal wait, evidence-download, and finish path; it cannot directly record success. Only verified evidence for that SHA can make `E2E / PR Gate` green. - **Two-phase ruleset rollout and backfill:** First deploy the E2E check producer and its trusted fork handling. Re-run `CI / Pull Request` for every already-open PR head/base pair (approving a first-time fork run when necessary), then verify that `E2E / PR Gate` is attached to that same head SHA for the current base. The fail-closed maintainer checker identifies any head still needing backfill. Adding the workflow does not retroactively report the context, and enabling the ruleset first leaves existing PRs at "Waiting for status to be reported." Do not activate the context as a security boundary while it is scoped only to the shared GitHub Actions app: that app identity cannot distinguish this workflow from another workflow. First give the gate distinct provenance through a dedicated GitHub App or an organization required-workflow rule. When the context is finally activated, also enable strict/up-to-date required status checks; otherwise a successful head can remain mergeable after `main` advances and changes the effective merge diff. Preserve the control-plane review and deterministic floor even after those external protections are in place. - **Contributor compliance failed:** Reject the PR and ask the contributor to provide the PR-body DCO declaration or replace unverified commits with a clean verified history. Do not approve, merge, amend, sign, or force-push on the contributor's behalf. - **Contributor/approver overlap:** Surface `advisories.contributorApprovalOverlap` when the same account not recognized as automated by the supported login conventions appears as the current PR opener, commit author, or co-author and its latest opinionated review is approved. The invalid state detected here is contributor and approver identity overlap in the current GitHub PR metadata; the source boundary is the current opener plus all commit-author and review pages fetched through GitHub's GraphQL API. The advisory includes contributors whose commits remain in the current PR head at check time; it does not retain original push actors or authors removed when history is rebased, squashed, or fixed up. A clear result is not proof of independent approval. Missing, invalid, or conflicting review timestamps, or failure to retrieve complete paginated history, produce a warning because the latest opinion cannot be selected reliably. diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml index dac06d6feae..2238b1f1219 100644 --- a/.github/workflows/e2e.yaml +++ b/.github/workflows/e2e.yaml @@ -144,6 +144,7 @@ jobs: - id: matrix name: Generate E2E target matrix env: + CHECKOUT_SHA: ${{ inputs.checkout_sha }} INFERENCE_MODE: ${{ inputs.inference_mode || 'mock' }} JOBS: ${{ inputs.jobs }} TARGETS: ${{ inputs.targets }} @@ -224,6 +225,14 @@ jobs: matrix="$(jq -c '.matrix' <<< "${plan}")" test_matrix="$(jq -c '.testMatrix' <<< "${plan}")" explicit_only_jobs_csv="$(jq -r '.explicitOnlyJobs | join(",")' <<< "${plan}")" + if [ -n "${CHECKOUT_SHA}" ]; then + expected_target_ids="$(jq -cn --arg targets "${TARGETS}" '$targets | if length == 0 then [] else split(",") end | sort')" + actual_target_ids="$(jq -c 'map(.id) | sort' <<< "${matrix}")" + if [ "${actual_target_ids}" != "${expected_target_ids}" ]; then + echo "::error::E2E planner matrix does not match controller-selected targets" >&2 + exit 1 + fi + fi echo "matrix=${matrix}" >> "$GITHUB_OUTPUT" echo "test_matrix=${test_matrix}" >> "$GITHUB_OUTPUT" echo "hermes_selected=${hermes_selected}" >> "$GITHUB_OUTPUT" diff --git a/test/e2e/README.md b/test/e2e/README.md index 7027968b4bf..c48c73134d6 100644 --- a/test/e2e/README.md +++ b/test/e2e/README.md @@ -177,8 +177,10 @@ out `checkout_sha`. The same validation verifies that the PR remains open, belongs to `NVIDIA/NemoClaw`, and still has both the dispatched head and base commits. The dispatch includes selected jobs, allowlisted typed targets, and valid plan and correlation metadata. Controller-bound targets are restricted -to the trusted allowlist; ordinary branch dispatch is not an acceptable -substitute. The controller uses GitHub's returned run ID for +to the trusted allowlist, and the trusted workflow requires the generated live +matrix IDs to exactly match that controller selection before credential-bearing +jobs can start. Ordinary branch dispatch is not an acceptable substitute. The +controller uses GitHub's returned run ID for waiting, evidence download, and completion, then revalidates that the PR is still open with the live head, base, and exact-diff coordination identity before recording a final result. The native observer revalidates the live revision diff --git a/test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts b/test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts index 8975e46ada9..c4915135a1d 100644 --- a/test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts +++ b/test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts @@ -42,7 +42,10 @@ function generateMatrixScript(): string { return String(step!.run); } -function executeGenerateMatrixWithPlannerOutput(plan: unknown) { +function executeGenerateMatrixWithPlannerOutput( + plan: unknown, + options: { checkoutSha?: string; jobs?: string; targets?: string } = {}, +) { const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-planner-schema-")); const binDirectory = path.join(directory, "bin"); const fakeNpx = path.join(binDirectory, "npx"); @@ -52,7 +55,12 @@ function executeGenerateMatrixWithPlannerOutput(plan: unknown) { fakeNpx, [ "#!/usr/bin/env bash", - '[[ "$#" -eq 4 && "$1" == "tsx" && "$2" == "tools/e2e/workflow-plan.mts" && "$3" == "--jobs" && "$4" == "cloud-onboard" ]] || exit 97', + "expected=(tsx tools/e2e/workflow-plan.mts)", + '[[ -z "${JOBS:-}" ]] || expected+=(--jobs "$JOBS")', + '[[ -z "${TARGETS:-}" ]] || expected+=(--targets "$TARGETS")', + 'actual=("$@")', + '[[ "${#actual[@]}" -eq "${#expected[@]}" ]] || exit 97', + 'for index in "${!expected[@]}"; do [[ "${actual[$index]}" == "${expected[$index]}" ]] || exit 97; done', "printf '%s\\n' \"${FAKE_E2E_PLAN}\"", "", ].join("\n"), @@ -65,13 +73,14 @@ function executeGenerateMatrixWithPlannerOutput(plan: unknown) { encoding: "utf8", env: { ...process.env, + CHECKOUT_SHA: options.checkoutSha ?? "", FAKE_E2E_PLAN: JSON.stringify(plan), GITHUB_OUTPUT: outputPath, GITHUB_STEP_SUMMARY: path.join(directory, "summary.md"), INFERENCE_MODE: "mock", - JOBS: "cloud-onboard", + JOBS: options.jobs ?? "cloud-onboard", PATH: `${binDirectory}${path.delimiter}${process.env.PATH ?? ""}`, - TARGETS: "", + TARGETS: options.targets ?? "", }, timeout: 30_000, }), @@ -780,6 +789,7 @@ it("carries the generated planner matrix through the workflow output and PR repo encoding: "utf8", env: { ...process.env, + CHECKOUT_SHA: "", GITHUB_OUTPUT: outputPath, GITHUB_STEP_SUMMARY: summaryPath, INFERENCE_MODE: "mock", @@ -844,6 +854,38 @@ it("fails closed when planner output violates the workflow schema", () => { } }); +it("binds controller matrix IDs to the trusted target selector (#7031)", () => { + const target = "ubuntu-repo-cloud-langchain-deepagents-code"; + const validPlan = buildE2eWorkflowPlan({ jobs: "cloud-onboard", targets: target }); + const options = { + checkoutSha: "a".repeat(40), + jobs: "cloud-onboard", + targets: target, + }; + + const matching = executeGenerateMatrixWithPlannerOutput(validPlan, options); + expect(matching.result.status, matching.result.stderr || matching.result.stdout).toBe(0); + + const injectedWithoutSelection = executeGenerateMatrixWithPlannerOutput(validPlan, { + ...options, + targets: "", + }); + expect(injectedWithoutSelection.result.status).toBe(1); + expect(injectedWithoutSelection.result.stderr).toContain( + "::error::E2E planner matrix does not match controller-selected targets", + ); + + const mismatchedPlan = { + ...validPlan, + matrix: validPlan.matrix.map((row) => ({ ...row, id: "ubuntu-repo-cloud-openclaw" })), + }; + const mismatched = executeGenerateMatrixWithPlannerOutput(mismatchedPlan, options); + expect(mismatched.result.status).toBe(1); + expect(mismatched.result.stderr).toContain( + "::error::E2E planner matrix does not match controller-selected targets", + ); +}); + it("requires the report-to-pr job to check out the trusted workflow revision", () => { const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "e2e-workflow-")); const workflowPath = path.join(tmp, "workflow.yaml"); diff --git a/test/e2e/support/e2e-workflow.test.ts b/test/e2e/support/e2e-workflow.test.ts index 2586cd84cb5..2e0899390ee 100644 --- a/test/e2e/support/e2e-workflow.test.ts +++ b/test/e2e/support/e2e-workflow.test.ts @@ -105,6 +105,30 @@ describe("e2e workflow boundary", () => { ); }); + it("keeps controller target selection bound to the generated matrix (#7031)", () => { + const workflow = readWorkflow() as { + jobs: Record< + string, + { steps?: Array<{ env?: Record; name?: string; run?: string }> } + >; + }; + const generate = workflow.jobs["generate-matrix"]?.steps?.find( + (step) => step.name === "Generate E2E target matrix", + )!; + delete generate.env!.CHECKOUT_SHA; + generate.run = generate.run!.replace( + "E2E planner matrix does not match controller-selected targets", + "unchecked planner matrix", + ); + + expect(validateE2eWorkflow(workflow)).toEqual( + expect.arrayContaining([ + "matrix generation step must bind controller checkout through CHECKOUT_SHA env", + "step 'Generate E2E target matrix' run script must include E2E planner matrix does not match controller-selected targets", + ]), + ); + }); + type RebuildWorkflowStep = { env?: Record; name?: string; diff --git a/test/pr-e2e-gate-typed-target.test.ts b/test/pr-e2e-gate-typed-target.test.ts index b4126abd99e..3dc9245504b 100644 --- a/test/pr-e2e-gate-typed-target.test.ts +++ b/test/pr-e2e-gate-typed-target.test.ts @@ -55,7 +55,7 @@ function signal(gate: PrGateState, overrides: Partial = {}): E2eR }; } -describe("PR E2E typed-target gate", () => { +describe("PR E2E typed-target gate (#7031)", () => { it("requires complete bound evidence for a target-only state", () => { const gate = state(); const target = signal(gate); diff --git a/test/pr-e2e-gate.test.ts b/test/pr-e2e-gate.test.ts index cb090175aeb..804be84b790 100644 --- a/test/pr-e2e-gate.test.ts +++ b/test/pr-e2e-gate.test.ts @@ -406,7 +406,7 @@ describe("PR E2E controller", () => { expect(() => expectedSignalShards(["not-a-workflow-job"])).toThrow(/does not define/u); }); - it("dispatches selected jobs and the allowlisted target with exact bound metadata", async () => { + it("dispatches selected jobs and the allowlisted target with exact bound metadata (#7031)", async () => { const jobs = ["onboard-repair", "onboard-resume", "full-e2e", "hermes-e2e"]; const fetchMock = vi.spyOn(globalThis, "fetch").mockImplementation( createGitHubFetchRouter([ diff --git a/tools/e2e/workflow-boundary.mts b/tools/e2e/workflow-boundary.mts index a83ff9706e8..db86f5d228e 100644 --- a/tools/e2e/workflow-boundary.mts +++ b/tools/e2e/workflow-boundary.mts @@ -3795,6 +3795,9 @@ export function validateE2eWorkflow(workflowValue: unknown): string[] { } const generate = requireStep(errors, generateSteps, "Generate E2E target matrix"); const generateEnv = asRecord(generate?.env); + if (generateEnv.CHECKOUT_SHA !== "${{ inputs.checkout_sha }}") { + errors.push("matrix generation step must bind controller checkout through CHECKOUT_SHA env"); + } if (generateEnv.JOBS !== "${{ inputs.jobs }}") { errors.push("matrix generation step must pass jobs through JOBS env"); } @@ -3824,6 +3827,14 @@ export function validateE2eWorkflow(workflowValue: unknown): string[] { requireRunContains(errors, generate, "expected_hermes_selected=false"); requireRunContains(errors, generate, "expected_hermes_selected=true"); requireRunContains(errors, generate, "E2E planner changed the trusted Hermes selection"); + requireRunContains(errors, generate, 'if [ -n "${CHECKOUT_SHA}" ]'); + requireRunContains(errors, generate, "expected_target_ids="); + requireRunContains(errors, generate, "actual_target_ids="); + requireRunContains( + errors, + generate, + "E2E planner matrix does not match controller-selected targets", + ); requireRunContains( errors, generate, From 7423f5fb3af5c89b2bbfe99b42dddb12b04daa93 Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Thu, 16 Jul 2026 13:58:53 -0700 Subject: [PATCH 3/7] test(e2e): use bash path separator Signed-off-by: Carlos Villela --- test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts b/test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts index c4915135a1d..9c26e05ee1b 100644 --- a/test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts +++ b/test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts @@ -79,7 +79,7 @@ function executeGenerateMatrixWithPlannerOutput( GITHUB_STEP_SUMMARY: path.join(directory, "summary.md"), INFERENCE_MODE: "mock", JOBS: options.jobs ?? "cloud-onboard", - PATH: `${binDirectory}${path.delimiter}${process.env.PATH ?? ""}`, + PATH: `${binDirectory}:${process.env.PATH ?? ""}`, TARGETS: options.targets ?? "", }, timeout: 30_000, From 2cc86ee96d4e955a3f79804a6e5875b84555b7fd Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Thu, 16 Jul 2026 14:53:20 -0700 Subject: [PATCH 4/7] fix(ci): pin controller target runner Signed-off-by: Charan Jagwani --- .github/workflows/e2e.yaml | 31 ++++++- test/e2e/README.md | 10 ++- ...e2e-report-to-pr-workflow-boundary.test.ts | 81 ++++++++++++++++++- test/e2e/support/e2e-workflow.test.ts | 31 +++++++ tools/e2e/workflow-boundary.mts | 56 ++++++++++++- 5 files changed, 195 insertions(+), 14 deletions(-) diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml index 2238b1f1219..f1ae7015bb8 100644 --- a/.github/workflows/e2e.yaml +++ b/.github/workflows/e2e.yaml @@ -87,11 +87,33 @@ jobs: generate-matrix: runs-on: ubuntu-latest outputs: - matrix: ${{ steps.matrix.outputs.matrix }} + matrix: ${{ steps.controller_matrix.outputs.matrix || steps.matrix.outputs.matrix }} test_matrix: ${{ steps.matrix.outputs.test_matrix }} hermes_selected: ${{ steps.matrix.outputs.hermes_selected }} explicit_only_jobs: ${{ steps.matrix.outputs.explicit_only_jobs }} steps: + - id: controller_matrix + name: Build trusted controller target matrix + if: ${{ inputs.checkout_sha != '' }} + env: + TARGETS: ${{ inputs.targets }} + shell: bash + run: | + set -euo pipefail + case "${TARGETS}" in + "") + matrix='[]' + ;; + ubuntu-repo-cloud-langchain-deepagents-code) + matrix='[{"id":"ubuntu-repo-cloud-langchain-deepagents-code","runner":"ubuntu-latest","label":"ubuntu-repo-cloud-langchain-deepagents-code"}]' + ;; + *) + echo "::error::PR E2E target is not approved by the trusted controller" >&2 + exit 1 + ;; + esac + printf 'matrix=%s\n' "${matrix}" >> "${GITHUB_OUTPUT}" + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 with: ref: ${{ inputs.checkout_sha || github.sha }} @@ -145,6 +167,7 @@ jobs: name: Generate E2E target matrix env: CHECKOUT_SHA: ${{ inputs.checkout_sha }} + CONTROLLER_MATRIX: ${{ steps.controller_matrix.outputs.matrix }} INFERENCE_MODE: ${{ inputs.inference_mode || 'mock' }} JOBS: ${{ inputs.jobs }} TARGETS: ${{ inputs.targets }} @@ -226,9 +249,9 @@ jobs: test_matrix="$(jq -c '.testMatrix' <<< "${plan}")" explicit_only_jobs_csv="$(jq -r '.explicitOnlyJobs | join(",")' <<< "${plan}")" if [ -n "${CHECKOUT_SHA}" ]; then - expected_target_ids="$(jq -cn --arg targets "${TARGETS}" '$targets | if length == 0 then [] else split(",") end | sort')" - actual_target_ids="$(jq -c 'map(.id) | sort' <<< "${matrix}")" - if [ "${actual_target_ids}" != "${expected_target_ids}" ]; then + expected_controller_matrix="$(jq -c 'map({id, runner}) | sort_by(.id)' <<< "${CONTROLLER_MATRIX}")" + actual_controller_matrix="$(jq -c 'map({id, runner}) | sort_by(.id)' <<< "${matrix}")" + if [ "${actual_controller_matrix}" != "${expected_controller_matrix}" ]; then echo "::error::E2E planner matrix does not match controller-selected targets" >&2 exit 1 fi diff --git a/test/e2e/README.md b/test/e2e/README.md index c48c73134d6..0dfe576a9c3 100644 --- a/test/e2e/README.md +++ b/test/e2e/README.md @@ -177,10 +177,12 @@ out `checkout_sha`. The same validation verifies that the PR remains open, belongs to `NVIDIA/NemoClaw`, and still has both the dispatched head and base commits. The dispatch includes selected jobs, allowlisted typed targets, and valid plan and correlation metadata. Controller-bound targets are restricted -to the trusted allowlist, and the trusted workflow requires the generated live -matrix IDs to exactly match that controller selection before credential-bearing -jobs can start. Ordinary branch dispatch is not an acceptable substitute. The -controller uses GitHub's returned run ID for +to the trusted allowlist. Before checking out PR code, the trusted workflow +projects each controller-selected target into a fixed target ID and hosted +runner mapping. The generated live matrix must exactly match those trusted IDs +and runners, and only the trusted projection can configure credential-bearing +typed-target jobs. Ordinary branch dispatch is not an acceptable substitute. +The controller uses GitHub's returned run ID for waiting, evidence download, and completion, then revalidates that the PR is still open with the live head, base, and exact-diff coordination identity before recording a final result. The native observer revalidates the live revision diff --git a/test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts b/test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts index 9c26e05ee1b..d574f076501 100644 --- a/test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts +++ b/test/e2e/support/e2e-report-to-pr-workflow-boundary.test.ts @@ -42,9 +42,48 @@ function generateMatrixScript(): string { return String(step!.run); } +function trustedControllerMatrixScript(): string { + const workflow = readWorkflow() as { + jobs: Record }>; + }; + const step = workflow.jobs["generate-matrix"].steps.find( + (candidate) => candidate.id === "controller_matrix", + ); + expect(step?.run).toEqual(expect.any(String)); + return String(step!.run); +} + +function executeTrustedControllerMatrix(targets: string) { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-controller-matrix-")); + const outputPath = path.join(directory, "github-output"); + try { + const result = spawnSync("bash", ["-c", trustedControllerMatrixScript()], { + cwd: process.cwd(), + encoding: "utf8", + env: { + ...process.env, + GITHUB_OUTPUT: outputPath, + TARGETS: targets, + }, + timeout: 30_000, + }); + return { + result, + workflowOutput: fs.existsSync(outputPath) ? fs.readFileSync(outputPath, "utf8") : "", + }; + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } +} + function executeGenerateMatrixWithPlannerOutput( plan: unknown, - options: { checkoutSha?: string; jobs?: string; targets?: string } = {}, + options: { + checkoutSha?: string; + controllerMatrix?: string; + jobs?: string; + targets?: string; + } = {}, ) { const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-planner-schema-")); const binDirectory = path.join(directory, "bin"); @@ -74,6 +113,7 @@ function executeGenerateMatrixWithPlannerOutput( env: { ...process.env, CHECKOUT_SHA: options.checkoutSha ?? "", + CONTROLLER_MATRIX: options.controllerMatrix ?? "", FAKE_E2E_PLAN: JSON.stringify(plan), GITHUB_OUTPUT: outputPath, GITHUB_STEP_SUMMARY: path.join(directory, "summary.md"), @@ -854,11 +894,36 @@ it("fails closed when planner output violates the workflow schema", () => { } }); -it("binds controller matrix IDs to the trusted target selector (#7031)", () => { +it("builds controller target matrices only from trusted runner mappings (#7031)", () => { + const target = "ubuntu-repo-cloud-langchain-deepagents-code"; + + const empty = executeTrustedControllerMatrix(""); + expect(empty.result.status, empty.result.stderr || empty.result.stdout).toBe(0); + expect(empty.workflowOutput).toBe("matrix=[]\n"); + + const approved = executeTrustedControllerMatrix(target); + expect(approved.result.status, approved.result.stderr || approved.result.stdout).toBe(0); + expect(parseSimpleOutput(approved.workflowOutput).matrix).toBe( + JSON.stringify([{ id: target, runner: "ubuntu-latest", label: target }]), + ); + + const rejected = executeTrustedControllerMatrix("untrusted-target"); + expect(rejected.result.status).toBe(1); + expect(rejected.result.stderr).toContain( + "::error::PR E2E target is not approved by the trusted controller", + ); + expect(rejected.workflowOutput).toBe(""); +}); + +it("binds controller matrix IDs and runners to the trusted target selector (#7031)", () => { const target = "ubuntu-repo-cloud-langchain-deepagents-code"; const validPlan = buildE2eWorkflowPlan({ jobs: "cloud-onboard", targets: target }); + const trustedControllerMatrix = JSON.stringify([ + { id: target, runner: "ubuntu-latest", label: target }, + ]); const options = { checkoutSha: "a".repeat(40), + controllerMatrix: trustedControllerMatrix, jobs: "cloud-onboard", targets: target, }; @@ -868,6 +933,7 @@ it("binds controller matrix IDs to the trusted target selector (#7031)", () => { const injectedWithoutSelection = executeGenerateMatrixWithPlannerOutput(validPlan, { ...options, + controllerMatrix: "[]", targets: "", }); expect(injectedWithoutSelection.result.status).toBe(1); @@ -884,6 +950,17 @@ it("binds controller matrix IDs to the trusted target selector (#7031)", () => { expect(mismatched.result.stderr).toContain( "::error::E2E planner matrix does not match controller-selected targets", ); + + const runnerInjectedPlan = { + ...validPlan, + matrix: validPlan.matrix.map((row) => ({ ...row, runner: "self-hosted" })), + }; + const runnerInjected = executeGenerateMatrixWithPlannerOutput(runnerInjectedPlan, options); + expect(runnerInjected.result.status).toBe(1); + expect(runnerInjected.result.stderr).toContain( + "::error::E2E planner matrix does not match controller-selected targets", + ); + expect(runnerInjected.workflowOutput).toBe(""); }); it("requires the report-to-pr job to check out the trusted workflow revision", () => { diff --git a/test/e2e/support/e2e-workflow.test.ts b/test/e2e/support/e2e-workflow.test.ts index 2e0899390ee..3e6d2843d91 100644 --- a/test/e2e/support/e2e-workflow.test.ts +++ b/test/e2e/support/e2e-workflow.test.ts @@ -129,6 +129,37 @@ describe("e2e workflow boundary", () => { ); }); + it("keeps controller runner selection in a trusted pre-checkout matrix (#7031)", () => { + const workflow = readWorkflow() as { + jobs: Record< + string, + { + outputs: Record; + steps: Array<{ id?: string; name?: string; run?: string; uses?: string }>; + } + >; + }; + const generateMatrix = workflow.jobs["generate-matrix"]!; + generateMatrix.outputs.matrix = "${{ steps.matrix.outputs.matrix }}"; + const trustedIndex = generateMatrix.steps.findIndex((step) => step.id === "controller_matrix"); + if (trustedIndex < 0) throw new Error("workflow missing trusted controller matrix step"); + const [trusted] = generateMatrix.steps.splice(trustedIndex, 1); + trusted!.run = trusted!.run!.replace('"runner":"ubuntu-latest"', '"runner":"self-hosted"'); + const checkoutIndex = generateMatrix.steps.findIndex((step) => + step.uses?.startsWith("actions/checkout@"), + ); + if (checkoutIndex < 0) throw new Error("workflow missing generate-matrix checkout step"); + generateMatrix.steps.splice(checkoutIndex + 1, 0, trusted!); + + expect(validateE2eWorkflow(workflow)).toEqual( + expect.arrayContaining([ + "generate-matrix job must expose trusted controller matrix output", + "trusted controller matrix must pin typed target runner to ubuntu-latest", + "trusted controller matrix step must run before PR checkout", + ]), + ); + }); + type RebuildWorkflowStep = { env?: Record; name?: string; diff --git a/tools/e2e/workflow-boundary.mts b/tools/e2e/workflow-boundary.mts index db86f5d228e..817d5906a4c 100644 --- a/tools/e2e/workflow-boundary.mts +++ b/tools/e2e/workflow-boundary.mts @@ -3771,8 +3771,11 @@ export function validateE2eWorkflow(workflowValue: unknown): string[] { errors.push("generate-matrix job must run on ubuntu-latest"); } const generateOutputs = asRecord(generateMatrix.outputs); - if (generateOutputs.matrix !== "${{ steps.matrix.outputs.matrix }}") { - errors.push("generate-matrix job must expose matrix output"); + if ( + generateOutputs.matrix !== + "${{ steps.controller_matrix.outputs.matrix || steps.matrix.outputs.matrix }}" + ) { + errors.push("generate-matrix job must expose trusted controller matrix output"); } if (generateOutputs.test_matrix !== "${{ steps.matrix.outputs.test_matrix }}") { errors.push("generate-matrix job must expose test_matrix output"); @@ -3785,10 +3788,52 @@ export function validateE2eWorkflow(workflowValue: unknown): string[] { } const generateSteps = asSteps(generateMatrix.steps); requireNoDispatchInputInterpolation(errors, generateSteps); + const controllerMatrix = requireJobStep( + errors, + "generate-matrix", + generateSteps, + "Build trusted controller target matrix", + ); + if (controllerMatrix?.id !== "controller_matrix") { + errors.push("trusted controller matrix step must use id controller_matrix"); + } + if (controllerMatrix?.if !== "${{ inputs.checkout_sha != '' }}") { + errors.push("trusted controller matrix step must run only for controller dispatches"); + } + if (controllerMatrix?.shell !== "bash") { + errors.push("trusted controller matrix step must use bash"); + } + const controllerMatrixEnv = asRecord(controllerMatrix?.env); + if (controllerMatrixEnv.TARGETS !== "${{ inputs.targets }}") { + errors.push("trusted controller matrix step must bind targets through TARGETS env"); + } + requireRunContains(errors, controllerMatrix, 'case "${TARGETS}" in'); + requireRunContains(errors, controllerMatrix, "matrix='[]'"); + requireRunContains(errors, controllerMatrix, "ubuntu-repo-cloud-langchain-deepagents-code"); + if (!stringValue(controllerMatrix?.run).includes('"runner":"ubuntu-latest"')) { + errors.push("trusted controller matrix must pin typed target runner to ubuntu-latest"); + } + requireRunContains( + errors, + controllerMatrix, + "PR E2E target is not approved by the trusted controller", + ); + requireRunContains( + errors, + controllerMatrix, + `printf 'matrix=%s\\n' "\${matrix}" >> "\${GITHUB_OUTPUT}"`, + ); const generateCheckout = generateSteps.find((step) => stringValue(step.uses).startsWith("actions/checkout@"), ); if (!generateCheckout) errors.push("generate-matrix job missing checkout step"); + if ( + controllerMatrix && + generateCheckout && + generateSteps.indexOf(controllerMatrix) >= generateSteps.indexOf(generateCheckout) + ) { + errors.push("trusted controller matrix step must run before PR checkout"); + } requireFullShaAction(errors, generateCheckout, "generate-matrix checkout"); if (asRecord(generateCheckout?.with)["persist-credentials"] !== false) { errors.push("generate-matrix checkout step must set persist-credentials=false"); @@ -3798,6 +3843,9 @@ export function validateE2eWorkflow(workflowValue: unknown): string[] { if (generateEnv.CHECKOUT_SHA !== "${{ inputs.checkout_sha }}") { errors.push("matrix generation step must bind controller checkout through CHECKOUT_SHA env"); } + if (generateEnv.CONTROLLER_MATRIX !== "${{ steps.controller_matrix.outputs.matrix }}") { + errors.push("matrix generation step must receive the trusted controller matrix"); + } if (generateEnv.JOBS !== "${{ inputs.jobs }}") { errors.push("matrix generation step must pass jobs through JOBS env"); } @@ -3828,8 +3876,8 @@ export function validateE2eWorkflow(workflowValue: unknown): string[] { requireRunContains(errors, generate, "expected_hermes_selected=true"); requireRunContains(errors, generate, "E2E planner changed the trusted Hermes selection"); requireRunContains(errors, generate, 'if [ -n "${CHECKOUT_SHA}" ]'); - requireRunContains(errors, generate, "expected_target_ids="); - requireRunContains(errors, generate, "actual_target_ids="); + requireRunContains(errors, generate, "expected_controller_matrix="); + requireRunContains(errors, generate, "actual_controller_matrix="); requireRunContains( errors, generate, From 322c3a29ad4406cef19911e471e24d07b20eb432 Mon Sep 17 00:00:00 2001 From: Apurv Kumaria Date: Thu, 16 Jul 2026 17:21:50 -0700 Subject: [PATCH 5/7] test(ci): keep typed-target fixture linear Co-authored-by: Carlos Villela Signed-off-by: Apurv Kumaria --- test/e2e/support/e2e-workflow.test.ts | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/test/e2e/support/e2e-workflow.test.ts b/test/e2e/support/e2e-workflow.test.ts index 3e6d2843d91..aa808a816f4 100644 --- a/test/e2e/support/e2e-workflow.test.ts +++ b/test/e2e/support/e2e-workflow.test.ts @@ -141,15 +141,16 @@ describe("e2e workflow boundary", () => { }; const generateMatrix = workflow.jobs["generate-matrix"]!; generateMatrix.outputs.matrix = "${{ steps.matrix.outputs.matrix }}"; - const trustedIndex = generateMatrix.steps.findIndex((step) => step.id === "controller_matrix"); - if (trustedIndex < 0) throw new Error("workflow missing trusted controller matrix step"); - const [trusted] = generateMatrix.steps.splice(trustedIndex, 1); + const [trusted] = generateMatrix.steps.splice( + generateMatrix.steps.findIndex((step) => step.id === "controller_matrix"), + 1, + ); trusted!.run = trusted!.run!.replace('"runner":"ubuntu-latest"', '"runner":"self-hosted"'); - const checkoutIndex = generateMatrix.steps.findIndex((step) => - step.uses?.startsWith("actions/checkout@"), + generateMatrix.steps.splice( + generateMatrix.steps.findIndex((step) => step.uses?.startsWith("actions/checkout@")) + 1, + 0, + trusted!, ); - if (checkoutIndex < 0) throw new Error("workflow missing generate-matrix checkout step"); - generateMatrix.steps.splice(checkoutIndex + 1, 0, trusted!); expect(validateE2eWorkflow(workflow)).toEqual( expect.arrayContaining([ From 43e53e75fa6fc0dd8f96a767698841907da2da21 Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Thu, 16 Jul 2026 20:45:50 -0700 Subject: [PATCH 6/7] chore(ci): revalidate Hermes recovery gate Signed-off-by: Carlos Villela From a16feee8da50741df54ec6ea7afc52a46e6af850 Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Thu, 16 Jul 2026 21:15:58 -0700 Subject: [PATCH 7/7] chore(ci): retry Hermes posture after runner shutdown Signed-off-by: Charan Jagwani