From 9d05ab37116291dd940639c8b1da284cd99f1ca7 Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Thu, 2 Jul 2026 14:19:15 -0700 Subject: [PATCH 1/3] docs(release): require exact-SHA E2E evidence Signed-off-by: Carlos Villela --- .../SKILL.md | 16 ++++++++-- .../PR-REVIEW-PRIORITIES.md | 4 +-- .../nemoclaw-maintainer-evening/SKILL.md | 3 +- .../references/daily-flow.md | 1 + .../references/release-train.md | 20 ++++++++++-- test/maintainer-skills-policy.test.ts | 31 +++++++++++++++++++ 6 files changed, 68 insertions(+), 7 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md index 3146ffce6ad..90f35a7e1ba 100644 --- a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md @@ -16,6 +16,7 @@ The release is one annotated semver tag on an already-merged `origin/main` commi ## Hard Rules - Tag only the commit captured in a generated release plan. +- Before asking for release confirmation, satisfy the canonical [pre-tag E2E evidence policy](../nemoclaw-maintainer-policies/references/release-train.md#pre-tag-e2e-evidence) for that exact commit. - Ask the maintainer to paste the exact confirmation phrase from the plan before cutting the tag. - Push only the semver tag (`vX.Y.Z`) from the agent-controlled step. - Never push `latest` or `lkg` from this skill. @@ -30,7 +31,7 @@ Copy this checklist and update it as you proceed: ```text Release Progress: - [ ] Step 1: Generate release plan -- [ ] Step 2: Show plan and exact confirmation phrase +- [ ] Step 2: Show plan, E2E evidence, and exact confirmation phrase - [ ] Step 3: Cut the semver tag from the confirmed plan - [ ] Step 4: Wait for workflow-managed latest - [ ] Step 5: Bump remaining open issues/PRs @@ -56,7 +57,7 @@ The script writes a plan outside the checkout root, for example: ../nemoclaw-release-v0.0.58/plan.json ``` -### Step 2: Show Plan and Ask for Exact Confirmation +### Step 2: Show Plan, E2E Evidence, and Ask for Exact Confirmation Read the generated `plan.json` and show the maintainer: @@ -68,6 +69,17 @@ Read the generated `plan.json` and show the maintainer: - exact confirmation phrase, - open issue/PR housekeeping plan for the release label. +For the plan's full `origin/main` SHA, review `.github/workflows/e2e.yaml` at that commit and build the evidence ledger required by the canonical [pre-tag E2E evidence policy](../nemoclaw-maintainer-policies/references/release-train.md#pre-tag-e2e-evidence). The workflow is the sole source of truth; do not substitute or maintain a separate release-gating test list. + +Before showing the confirmation prompt, present: + +- the exact candidate SHA; +- the number of tests with green evidence out of the number required by the workflow; +- each required test mapped to a successful run or job URL and attempt; and +- an itemized maintainer exception for every test without green evidence, including its current result or failure summary and the rationale for proceeding. + +Do not ask for the exact phrase until every test has green evidence or an explicit itemized maintainer exception. If `origin/main` moves or the candidate SHA otherwise changes, regenerate the plan and rebuild the ledger for the new SHA. + Ask the maintainer to paste the exact phrase: ```text diff --git a/.agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md b/.agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md index 2d497b070fe..de9e15bb159 100644 --- a/.agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md +++ b/.agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md @@ -34,8 +34,8 @@ The team follows a daily ship cycle. All maintainer skills operate within this r 1. **Morning** (`/nemoclaw-maintainer-morning`) — triage the backlog, pick items for the day, label them with the target version (e.g., `v0.0.8`). 2. **During the day** (`/nemoclaw-maintainer-day`) — land PRs using the maintainer loop. Version labels make progress visible on dashboards. -3. **Evening** (`/nemoclaw-maintainer-evening`) — check what shipped, identify open stragglers, generate a QA-focused summary, cut the tag, automatically bump stragglers to the next patch, and prepare release notes for posting. -4. **Overnight** — QA team (different timezone) tests the tag. Any issues they file enter the next morning's triage like any other issue. +3. **Evening** (`/nemoclaw-maintainer-evening`) — check what shipped, identify open stragglers, generate a QA-focused summary, freeze the exact candidate SHA, collect the E2E evidence or itemized maintainer exceptions required before confirmation, cut the tag, automatically bump stragglers to the next patch, and prepare release notes for posting. +4. **Overnight** — QA team (different timezone) performs additional validation of the tag. Any issues they file enter the next morning's triage like any other issue. Version labels activate release work; they are not readiness claims. If an open item misses the tag, its label moves to the next patch during post-tag housekeeping. diff --git a/.agents/skills/nemoclaw-maintainer-evening/SKILL.md b/.agents/skills/nemoclaw-maintainer-evening/SKILL.md index 4e2902d2a37..5e77316eead 100644 --- a/.agents/skills/nemoclaw-maintainer-evening/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-evening/SKILL.md @@ -42,13 +42,14 @@ This lists commits since the last tag, identifies risky areas touched, and sugge ## Step 4: Cut the Tag and Publish Release Notes -Load `cut-release-tag`. The version is already known — default to patch bump, but still show the commit, changelog, post-tag bump plan, and release notes draft for confirmation. NemoClaw releases are tag-based: tag `main`, let the workflow move `latest`, automatically bump remaining open issues/PRs to the next patch label, and prepare the release notes announcement for the maintainer to post. +Load `cut-release-tag`. The version is already known — default to patch bump, but still show the commit, changelog, post-tag bump plan, and release notes draft for confirmation. After the release plan freezes the exact candidate SHA, review the pre-tag E2E evidence ledger derived from `.github/workflows/e2e.yaml` at that commit. Do not ask for the release confirmation phrase until every test has green evidence or an explicit itemized maintainer exception. NemoClaw releases are tag-based: tag `main`, let the workflow move `latest`, automatically bump remaining open issues/PRs to the next patch label, and prepare the release notes announcement for the maintainer to post. ## Step 5: Confirm and Share After the tag is cut and release notes are drafted or posted by the maintainer, present the final summary: - **Tag**: `v0.0.8` at commit `abc1234` +- **Pre-tag E2E evidence**: 12/13 tests green for the exact candidate SHA; 1 itemized maintainer exception - **Release notes draft**: `../nemoclaw-release-v0.0.8/release-note-draft.md` - **Shipped**: 4 items (#1234, #1235, #1236, #1237) - **Bumped to v0.0.9**: 1 item (#1238 — still needs CI fix) diff --git a/.agents/skills/nemoclaw-maintainer-policies/references/daily-flow.md b/.agents/skills/nemoclaw-maintainer-policies/references/daily-flow.md index eec60554251..76f6b11b12c 100644 --- a/.agents/skills/nemoclaw-maintainer-policies/references/daily-flow.md +++ b/.agents/skills/nemoclaw-maintainer-policies/references/daily-flow.md @@ -68,5 +68,6 @@ Agents may recommend labels, assignments, Project field changes, comments, merge - A PR daily version label activates daily release work; it is not a readiness claim. - Release inclusion requires a PR to be both merged and carrying the relevant daily version label at release cutoff. - Issue daily version labels are tracking or coordination signals only. +- Before tag confirmation, freeze the exact candidate SHA and review every E2E test declared by `.github/workflows/e2e.yaml` at that commit. Each test needs green evidence for that SHA or an explicit itemized maintainer exception. - Open PRs and issues that miss a tagged release carry forward by automatically moving from the released version label to the next patch label after the tag and `latest` are verified. - Durable release history belongs in releases, release notes, or manifests, not in long-lived labels. diff --git a/.agents/skills/nemoclaw-maintainer-policies/references/release-train.md b/.agents/skills/nemoclaw-maintainer-policies/references/release-train.md index 750880212ed..6cdb8bd6c5c 100644 --- a/.agents/skills/nemoclaw-maintainer-policies/references/release-train.md +++ b/.agents/skills/nemoclaw-maintainer-policies/references/release-train.md @@ -27,8 +27,24 @@ At cutoff: 2. Confirm each is intended for the release. 3. List open PRs and issues still carrying the target label as post-tag stragglers. 4. Generate QA handoff from merged PRs. -5. Cut the release tag only with explicit maintainer confirmation. -6. After the tag and workflow-managed `latest` are verified, automatically move every open straggler to the next patch label. +5. Generate the release plan to freeze the exact candidate commit. +6. Review the candidate commit's pre-tag E2E evidence. +7. Cut the release tag only with explicit maintainer confirmation. +8. After the tag and workflow-managed `latest` are verified, automatically move every open straggler to the next patch label. + +## Pre-Tag E2E Evidence + +The release candidate is the exact full `origin/main` commit SHA captured by the generated release plan. At that commit, `.github/workflows/e2e.yaml` is the sole source of truth for the release E2E test set. Do not maintain a separate release-gating test list. + +Before asking for the exact release confirmation phrase, build and show an evidence ledger for that SHA: + +- Every E2E test execution declared by the workflow must have at least one completed, successful execution for the candidate SHA. This includes tests that require explicit selection and every expanded matrix execution. +- Green evidence may accumulate across multiple workflow runs, selective runs, reruns, and attempts. A later failure does not erase an earlier successful execution for the same test and SHA. +- Skipped, unexecuted, queued, in-progress, cancelled, and failing results are not green evidence. +- Map each test with green evidence to its successful run or job URL and attempt number. +- If a test has no successful execution, the tag may still proceed at maintainer discretion only with an itemized maintainer exception that records the test identifier, relevant run links or available evidence, the current result or failure summary, and the rationale for proceeding. + +Every test must have either green evidence or an itemized maintainer exception before the release confirmation is requested. If the candidate SHA changes, discard the ledger and its exceptions, regenerate the release plan, and repeat the review for the new SHA. ## Carry Forward diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index eebfab8dc28..95875f5a64a 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -86,6 +86,37 @@ describe("maintainer skills follow canonical workflow policy", () => { ).toBe(true); }); + it("requires exact-SHA E2E evidence or itemized maintainer exceptions before tagging", () => { + const evening = read(".agents/skills/nemoclaw-maintainer-evening/SKILL.md"); + const release = read(".agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md"); + const policy = read(".agents/skills/nemoclaw-maintainer-policies/references/release-train.md"); + + expect(policy).toContain("exact full `origin/main` commit SHA"); + expect(policy).toContain("`.github/workflows/e2e.yaml` is the sole source of truth"); + expect(policy).toContain("Do not maintain a separate release-gating test list"); + expect(policy).toContain("at least one completed, successful execution"); + expect(policy).toContain("multiple workflow runs, selective runs, reruns, and attempts"); + expect(policy).toContain("explicit selection and every expanded matrix execution"); + expect(policy).toContain("A later failure does not erase an earlier successful execution"); + expect(policy).toContain( + "Skipped, unexecuted, queued, in-progress, cancelled, and failing results are not green evidence", + ); + expect(policy).toContain("itemized maintainer exception"); + expect(policy).toContain("If the candidate SHA changes"); + expect(policy).toContain("discard the ledger and its exceptions"); + expect(release).toContain("the number of tests with green evidence"); + expect(release).toContain("successful run or job URL and attempt"); + const evidenceSummary = release.indexOf("Before showing the confirmation prompt"); + const confirmationPrompt = release.indexOf( + "Ask the maintainer to paste the exact phrase", + evidenceSummary, + ); + expect(evidenceSummary).toBeGreaterThanOrEqual(0); + expect(evidenceSummary).toBeLessThan(confirmationPrompt); + expect(evening).toContain("every test has green evidence"); + expect(evening).toContain("explicit itemized maintainer exception"); + }); + it("keeps cross-issue sweeping separate from comparator scoring", () => { const sweep = read(".agents/skills/nemoclaw-maintainer-cross-issue-sweep/SKILL.md"); const comparator = read(".agents/skills/nemoclaw-maintainer-pr-comparator/SKILL.md"); From 86b4cb0fb60d7af5a09329299dfed01373dc2fac Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Thu, 2 Jul 2026 14:41:21 -0700 Subject: [PATCH 2/3] docs(release): clarify matrix evidence entries Signed-off-by: Carlos Villela --- .../references/release-train.md | 1 + test/maintainer-skills-policy.test.ts | 6 ++++++ 2 files changed, 7 insertions(+) diff --git a/.agents/skills/nemoclaw-maintainer-policies/references/release-train.md b/.agents/skills/nemoclaw-maintainer-policies/references/release-train.md index 6cdb8bd6c5c..941d825f6d0 100644 --- a/.agents/skills/nemoclaw-maintainer-policies/references/release-train.md +++ b/.agents/skills/nemoclaw-maintainer-policies/references/release-train.md @@ -39,6 +39,7 @@ The release candidate is the exact full `origin/main` commit SHA captured by the Before asking for the exact release confirmation phrase, build and show an evidence ledger for that SHA: - Every E2E test execution declared by the workflow must have at least one completed, successful execution for the candidate SHA. This includes tests that require explicit selection and every expanded matrix execution. +- Treat each expanded matrix execution as a separate ledger entry. Use its matrix `id`, or all distinguishing matrix dimensions when no single ID exists, in the test identifier so results for distinct expansions are never collapsed under the parent job. - Green evidence may accumulate across multiple workflow runs, selective runs, reruns, and attempts. A later failure does not erase an earlier successful execution for the same test and SHA. - Skipped, unexecuted, queued, in-progress, cancelled, and failing results are not green evidence. - Map each test with green evidence to its successful run or job URL and attempt number. diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index 95875f5a64a..62893fdb8ec 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -87,7 +87,9 @@ describe("maintainer skills follow canonical workflow policy", () => { }); it("requires exact-SHA E2E evidence or itemized maintainer exceptions before tagging", () => { + const dailyFlow = read(".agents/skills/nemoclaw-maintainer-policies/references/daily-flow.md"); const evening = read(".agents/skills/nemoclaw-maintainer-evening/SKILL.md"); + const priorities = read(".agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md"); const release = read(".agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md"); const policy = read(".agents/skills/nemoclaw-maintainer-policies/references/release-train.md"); @@ -97,6 +99,8 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(policy).toContain("at least one completed, successful execution"); expect(policy).toContain("multiple workflow runs, selective runs, reruns, and attempts"); expect(policy).toContain("explicit selection and every expanded matrix execution"); + expect(policy).toContain("each expanded matrix execution as a separate ledger entry"); + expect(policy).toContain("matrix `id`"); expect(policy).toContain("A later failure does not erase an earlier successful execution"); expect(policy).toContain( "Skipped, unexecuted, queued, in-progress, cancelled, and failing results are not green evidence", @@ -115,6 +119,8 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(evidenceSummary).toBeLessThan(confirmationPrompt); expect(evening).toContain("every test has green evidence"); expect(evening).toContain("explicit itemized maintainer exception"); + expect(dailyFlow).toContain("freeze the exact candidate SHA and review every E2E test"); + expect(priorities).toContain("collect the E2E evidence or itemized maintainer exceptions"); }); it("keeps cross-issue sweeping separate from comparator scoring", () => { From 399560e17e96b42a2765441b1381ca2643c69320 Mon Sep 17 00:00:00 2001 From: Carlos Villela Date: Thu, 2 Jul 2026 14:58:13 -0700 Subject: [PATCH 3/3] docs(release): name the confirmed tag target Signed-off-by: Carlos Villela --- .agents/skills/nemoclaw-maintainer-evening/SKILL.md | 2 +- test/maintainer-skills-policy.test.ts | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.agents/skills/nemoclaw-maintainer-evening/SKILL.md b/.agents/skills/nemoclaw-maintainer-evening/SKILL.md index 5e77316eead..a62786e6484 100644 --- a/.agents/skills/nemoclaw-maintainer-evening/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-evening/SKILL.md @@ -42,7 +42,7 @@ This lists commits since the last tag, identifies risky areas touched, and sugge ## Step 4: Cut the Tag and Publish Release Notes -Load `cut-release-tag`. The version is already known — default to patch bump, but still show the commit, changelog, post-tag bump plan, and release notes draft for confirmation. After the release plan freezes the exact candidate SHA, review the pre-tag E2E evidence ledger derived from `.github/workflows/e2e.yaml` at that commit. Do not ask for the release confirmation phrase until every test has green evidence or an explicit itemized maintainer exception. NemoClaw releases are tag-based: tag `main`, let the workflow move `latest`, automatically bump remaining open issues/PRs to the next patch label, and prepare the release notes announcement for the maintainer to post. +Load `cut-release-tag`. The version is already known — default to patch bump, but still show the commit, changelog, post-tag bump plan, and release notes draft for confirmation. After the release plan freezes the exact candidate SHA, review the pre-tag E2E evidence ledger derived from `.github/workflows/e2e.yaml` at that commit. Do not ask for the release confirmation phrase until every test has green evidence or an explicit itemized maintainer exception. NemoClaw releases are tag-based: tag the confirmed release commit with `vX.Y.Z`, let the workflow move `latest`, automatically bump remaining open issues/PRs to the next patch label, and prepare the release notes announcement for the maintainer to post. ## Step 5: Confirm and Share diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index 62893fdb8ec..16f13cd8edd 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -119,6 +119,8 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(evidenceSummary).toBeLessThan(confirmationPrompt); expect(evening).toContain("every test has green evidence"); expect(evening).toContain("explicit itemized maintainer exception"); + expect(evening).toContain("tag the confirmed release commit with `vX.Y.Z`"); + expect(evening).not.toContain("tag `main`"); expect(dailyFlow).toContain("freeze the exact candidate SHA and review every E2E test"); expect(priorities).toContain("collect the E2E evidence or itemized maintainer exceptions"); });