diff --git a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md index e99c604afe4..20f85832724 100644 --- a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md @@ -18,6 +18,7 @@ The release is one annotated semver tag on an already-merged `origin/main` commi - Tag only the commit captured in a generated release plan. - Do not generate the release plan until release-prep docs are merged or explicitly waived. - If `origin/main` changes after plan generation, regenerate the plan before cutting the tag. +- Before asking for release confirmation, satisfy the canonical [pre-tag E2E evidence policy](../nemoclaw-maintainer-policies/references/release-train.md#pre-tag-e2e-evidence) for that exact commit. - Ask the maintainer to paste the exact confirmation phrase from the plan before cutting the tag. - Push only the semver tag (`vX.Y.Z`) from the agent-controlled step. - Never push `latest` or `lkg` from this skill. @@ -32,7 +33,7 @@ Copy this checklist and update it as you proceed: ```text Release Progress: - [ ] Step 1: Generate release plan -- [ ] Step 2: Show plan and exact confirmation phrase +- [ ] Step 2: Show plan, E2E evidence, and exact confirmation phrase - [ ] Step 3: Cut the semver tag from the confirmed plan - [ ] Step 4: Wait for workflow-managed latest - [ ] Step 5: Bump remaining open issues/PRs @@ -61,7 +62,7 @@ The script writes a plan outside the checkout root, for example: ../nemoclaw-release-v0.0.58/plan.json ``` -### Step 2: Show Plan and Ask for Exact Confirmation +### Step 2: Show Plan, E2E Evidence, and Ask for Exact Confirmation Read the generated `plan.json` and show the maintainer: @@ -73,6 +74,17 @@ Read the generated `plan.json` and show the maintainer: - exact confirmation phrase, - open issue/PR housekeeping plan for the release label. +For the plan's full `origin/main` SHA, review `.github/workflows/e2e.yaml` at that commit and build the evidence ledger required by the canonical [pre-tag E2E evidence policy](../nemoclaw-maintainer-policies/references/release-train.md#pre-tag-e2e-evidence). The workflow is the sole source of truth; do not substitute or maintain a separate release-gating test list. + +Before showing the confirmation prompt, present: + +- the exact candidate SHA; +- the number of tests with green evidence out of the number required by the workflow; +- each required test mapped to a successful run or job URL and attempt; and +- an itemized maintainer exception for every test without green evidence, including its current result or failure summary and the rationale for proceeding. + +Do not ask for the exact phrase until every test has green evidence or an explicit itemized maintainer exception. If `origin/main` moves or the candidate SHA otherwise changes, regenerate the plan and rebuild the ledger for the new SHA. + Ask the maintainer to paste the exact phrase: ```text diff --git a/.agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md b/.agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md index 2d497b070fe..de9e15bb159 100644 --- a/.agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md +++ b/.agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md @@ -34,8 +34,8 @@ The team follows a daily ship cycle. All maintainer skills operate within this r 1. **Morning** (`/nemoclaw-maintainer-morning`) — triage the backlog, pick items for the day, label them with the target version (e.g., `v0.0.8`). 2. **During the day** (`/nemoclaw-maintainer-day`) — land PRs using the maintainer loop. Version labels make progress visible on dashboards. -3. **Evening** (`/nemoclaw-maintainer-evening`) — check what shipped, identify open stragglers, generate a QA-focused summary, cut the tag, automatically bump stragglers to the next patch, and prepare release notes for posting. -4. **Overnight** — QA team (different timezone) tests the tag. Any issues they file enter the next morning's triage like any other issue. +3. **Evening** (`/nemoclaw-maintainer-evening`) — check what shipped, identify open stragglers, generate a QA-focused summary, freeze the exact candidate SHA, collect the E2E evidence or itemized maintainer exceptions required before confirmation, cut the tag, automatically bump stragglers to the next patch, and prepare release notes for posting. +4. **Overnight** — QA team (different timezone) performs additional validation of the tag. Any issues they file enter the next morning's triage like any other issue. Version labels activate release work; they are not readiness claims. If an open item misses the tag, its label moves to the next patch during post-tag housekeeping. diff --git a/.agents/skills/nemoclaw-maintainer-evening/SKILL.md b/.agents/skills/nemoclaw-maintainer-evening/SKILL.md index 4a2833e5486..a96f53d87ca 100644 --- a/.agents/skills/nemoclaw-maintainer-evening/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-evening/SKILL.md @@ -48,13 +48,14 @@ If a docs PR or any other intended PR merges after `release:plan`, regenerate th ## Step 4: Cut the Tag and Publish Release Notes -Load `cut-release-tag`. The version is already known — default to patch bump, but still show the commit, changelog, post-tag bump plan, and release notes draft for confirmation. NemoClaw releases are tag-based: tag `main`, let the workflow move `latest`, automatically bump remaining open issues/PRs to the next patch label, and prepare the release notes announcement for the maintainer to post. +Load `cut-release-tag`. The version is already known — default to patch bump, but still show the commit, changelog, post-tag bump plan, and release notes draft for confirmation. After the release plan freezes the exact candidate SHA, review the pre-tag E2E evidence ledger derived from `.github/workflows/e2e.yaml` at that commit. Do not ask for the release confirmation phrase until every test has green evidence or an explicit itemized maintainer exception. NemoClaw releases are tag-based: tag the confirmed release commit with `vX.Y.Z`, let the workflow move `latest`, automatically bump remaining open issues/PRs to the next patch label, and prepare the release notes announcement for the maintainer to post. ## Step 5: Confirm and Share After the tag is cut and release notes are drafted or posted by the maintainer, present the final summary: - **Tag**: `v0.0.8` at commit `abc1234` +- **Pre-tag E2E evidence**: 12/13 tests green for the exact candidate SHA; 1 itemized maintainer exception - **Release notes draft**: `../nemoclaw-release-v0.0.8/release-note-draft.md` - **Shipped**: 4 items (#1234, #1235, #1236, #1237) - **Bumped to v0.0.9**: 1 item (#1238 — still needs CI fix) diff --git a/.agents/skills/nemoclaw-maintainer-policies/references/daily-flow.md b/.agents/skills/nemoclaw-maintainer-policies/references/daily-flow.md index eec60554251..76f6b11b12c 100644 --- a/.agents/skills/nemoclaw-maintainer-policies/references/daily-flow.md +++ b/.agents/skills/nemoclaw-maintainer-policies/references/daily-flow.md @@ -68,5 +68,6 @@ Agents may recommend labels, assignments, Project field changes, comments, merge - A PR daily version label activates daily release work; it is not a readiness claim. - Release inclusion requires a PR to be both merged and carrying the relevant daily version label at release cutoff. - Issue daily version labels are tracking or coordination signals only. +- Before tag confirmation, freeze the exact candidate SHA and review every E2E test declared by `.github/workflows/e2e.yaml` at that commit. Each test needs green evidence for that SHA or an explicit itemized maintainer exception. - Open PRs and issues that miss a tagged release carry forward by automatically moving from the released version label to the next patch label after the tag and `latest` are verified. - Durable release history belongs in releases, release notes, or manifests, not in long-lived labels. diff --git a/.agents/skills/nemoclaw-maintainer-policies/references/release-train.md b/.agents/skills/nemoclaw-maintainer-policies/references/release-train.md index 3b83dc9c52b..74bc8707161 100644 --- a/.agents/skills/nemoclaw-maintainer-policies/references/release-train.md +++ b/.agents/skills/nemoclaw-maintainer-policies/references/release-train.md @@ -33,8 +33,25 @@ At cutoff: 2. Confirm each is intended for the release. 3. List open PRs and issues still carrying the target label as post-tag stragglers. 4. Generate QA handoff from merged PRs. -5. Cut the release tag only with explicit maintainer confirmation. -6. After the tag and workflow-managed `latest` are verified, automatically move every open straggler to the next patch label. +5. Generate the release plan to freeze the exact candidate commit. +6. Review the candidate commit's pre-tag E2E evidence. +7. Cut the release tag only with explicit maintainer confirmation. +8. After the tag and workflow-managed `latest` are verified, automatically move every open straggler to the next patch label. + +## Pre-Tag E2E Evidence + +The release candidate is the exact full `origin/main` commit SHA captured by the generated release plan. At that commit, `.github/workflows/e2e.yaml` is the sole source of truth for the release E2E test set. Do not maintain a separate release-gating test list. + +Before asking for the exact release confirmation phrase, build and show an evidence ledger for that SHA: + +- Every E2E test execution declared by the workflow must have at least one completed, successful execution for the candidate SHA. This includes tests that require explicit selection and every expanded matrix execution. +- Treat each expanded matrix execution as a separate ledger entry. Use its matrix `id`, or all distinguishing matrix dimensions when no single ID exists, in the test identifier so results for distinct expansions are never collapsed under the parent job. +- Green evidence may accumulate across multiple workflow runs, selective runs, reruns, and attempts. A later failure does not erase an earlier successful execution for the same test and SHA. +- Skipped, unexecuted, queued, in-progress, cancelled, and failing results are not green evidence. +- Map each test with green evidence to its successful run or job URL and attempt number. +- If a test has no successful execution, the tag may still proceed at maintainer discretion only with an itemized maintainer exception that records the test identifier, relevant run links or available evidence, the current result or failure summary, and the rationale for proceeding. + +Every test must have either green evidence or an itemized maintainer exception before the release confirmation is requested. If the candidate SHA changes, discard the ledger and its exceptions, regenerate the release plan, and repeat the review for the new SHA. ## Carry Forward diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index 79bd0938d5d..127c3c64bd5 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -86,6 +86,45 @@ describe("maintainer skills follow canonical workflow policy", () => { ).toBe(true); }); + it("requires exact-SHA E2E evidence or itemized maintainer exceptions before tagging", () => { + const dailyFlow = read(".agents/skills/nemoclaw-maintainer-policies/references/daily-flow.md"); + const evening = read(".agents/skills/nemoclaw-maintainer-evening/SKILL.md"); + const priorities = read(".agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md"); + const release = read(".agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md"); + const policy = read(".agents/skills/nemoclaw-maintainer-policies/references/release-train.md"); + + expect(policy).toContain("exact full `origin/main` commit SHA"); + expect(policy).toContain("`.github/workflows/e2e.yaml` is the sole source of truth"); + expect(policy).toContain("Do not maintain a separate release-gating test list"); + expect(policy).toContain("at least one completed, successful execution"); + expect(policy).toContain("multiple workflow runs, selective runs, reruns, and attempts"); + expect(policy).toContain("explicit selection and every expanded matrix execution"); + expect(policy).toContain("each expanded matrix execution as a separate ledger entry"); + expect(policy).toContain("matrix `id`"); + expect(policy).toContain("A later failure does not erase an earlier successful execution"); + expect(policy).toContain( + "Skipped, unexecuted, queued, in-progress, cancelled, and failing results are not green evidence", + ); + expect(policy).toContain("itemized maintainer exception"); + expect(policy).toContain("If the candidate SHA changes"); + expect(policy).toContain("discard the ledger and its exceptions"); + expect(release).toContain("the number of tests with green evidence"); + expect(release).toContain("successful run or job URL and attempt"); + const evidenceSummary = release.indexOf("Before showing the confirmation prompt"); + const confirmationPrompt = release.indexOf( + "Ask the maintainer to paste the exact phrase", + evidenceSummary, + ); + expect(evidenceSummary).toBeGreaterThanOrEqual(0); + expect(evidenceSummary).toBeLessThan(confirmationPrompt); + expect(evening).toContain("every test has green evidence"); + expect(evening).toContain("explicit itemized maintainer exception"); + expect(evening).toContain("tag the confirmed release commit with `vX.Y.Z`"); + expect(evening).not.toContain("tag `main`"); + expect(dailyFlow).toContain("freeze the exact candidate SHA and review every E2E test"); + expect(priorities).toContain("collect the E2E evidence or itemized maintainer exceptions"); + }); + it("runs release-prep docs before generating the final release plan", () => { const updateDocs = read(".agents/skills/nemoclaw-contributor-update-docs/SKILL.md"); const evening = read(".agents/skills/nemoclaw-maintainer-evening/SKILL.md");