Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
394 changes: 80 additions & 314 deletions .agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md

Large diffs are not rendered by default.

This file was deleted.

16 changes: 7 additions & 9 deletions .agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,15 +53,13 @@ write a suggestion.

The team follows a daily ship cycle. All maintainer skills operate within this rhythm.

1. **Morning** (`/nemoclaw-maintainer-morning`) — triage the backlog, pick items for the day, label them with the target version (e.g., `v0.0.8`).
2. **During the day** (`/nemoclaw-maintainer-day`) — land PRs using the maintainer loop. Version labels make progress visible on dashboards.
3. **Evening** (`/nemoclaw-maintainer-evening`) — Check shipped work and the pre-tag changelog PR.
Confirm that the pre-tag changelog PR contains `docs/changelog/YYYY-MM-DD.mdx` for the release.
Identify open items and prepare the QA summary. Record the release SHA and required E2E evidence.
Cut the tag after confirmation. Move open items to the next patch label and delete the released label.
Prepare the Announcement.
4. **Overnight** — A QA team in another time zone validates the tag.
Put new issues into the next morning's triage.
1. **Morning** (`/nemoclaw-maintainer-morning`): Triage the backlog, pick items for the day, and label them with the target version (for example, `v0.0.8`).
2. **During the day** (`/nemoclaw-maintainer-day`): Land PRs using the maintainer loop. Version labels make progress visible on dashboards.
3. **4 PM close** (`/nemoclaw-maintainer-evening`): Stop merging, require the dated changelog entry, freeze the exact candidate, identify stragglers, and start the advisory overnight handoff.
4. **4 PM–4 AM frozen loop**: Use exact-SHA post-merge agent reviews and the E2E runs from every edition `main` push to diagnose regressions, rerun selectively, and prepare fixes without merging.
5. **4 AM cut** (`/nemoclaw-maintainer-cut-release-tag`): Tag the frozen candidate regardless of E2E state. Keep changelog, ancestry, tag, signing, `latest`, `lkg`, and housekeeping controls fail-closed.
6. **4 AM–8 AM continuation**: Continue advisory diagnosis and fix preparation.
7. **8 AM handoff** (`/nemoclaw-maintainer-morning`): Hand state to the next release doula and reopen the merge window.

Version labels activate release work. They do not show readiness.
If an open item misses the tag, move its label to the next patch after the release.
Expand Down
80 changes: 61 additions & 19 deletions .agents/skills/nemoclaw-maintainer-e2e/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: nemoclaw-maintainer-e2e
description: Dispatches and verifies trusted GitHub Actions E2E for NemoClaw maintainers, including manual PR E2E for the current PR head commit. Use for requests such as run E2E for PR #123, run the E2E suite, run the Launchable E2E, run the full E2E suite, deploy pre-release full E2E, run pre-tag full E2E, or run release-candidate E2E.
description: Dispatches and verifies trusted advisory GitHub Actions E2E for NemoClaw maintainers, including exact-revision manual PR E2E and per-main-push overnight diagnosis. Use for requests such as run E2E for PR #123, run the E2E suite, diagnose post-merge E2E, selectively rerun failures, or validate a release candidate without gating its tag.
---

<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
Expand All @@ -9,7 +9,7 @@ description: Dispatches and verifies trusted GitHub Actions E2E for NemoClaw mai
# Run Maintainer E2E

Use `.github/workflows/e2e.yaml` from trusted `main`.
Every push to `main` selects every workflow E2E. A selected job can remain queued until its configured runner is available. No E2E job is excluded from trusted `main` push selection. Pre-tag evidence still requires the full `workflow_dispatch` mode described below.
Every push to `main` selects every workflow E2E. A selected job can remain queued until its configured runner is available. No workflow E2E requires a separate explicit dispatch.
Do not substitute local `npm run test:live-e2e` unless the maintainer explicitly requests local execution.

## Manual PR E2E
Expand Down Expand Up @@ -144,8 +144,7 @@ A changed head repository, head SHA, or base SHA invalidates the evidence and re
| “Run the E2E suite” | Ordinary | empty | `false` |
| “Run the Launchable E2E” | Launchable | `staging-brev-launchable` | `false` |
| “Run the full E2E suite” | Full | empty | `true` |
| “deploy pre-release full E2E” | Full | empty | `true` |
| “run pre-tag full E2E” | Full | empty | `true` |
| “diagnose post-merge E2E” | Main-push inspection | n/a | n/a |
| “run release-candidate E2E” | Full | empty | `true` |

A generic E2E request must not authorize the Brev Launchable path.
Expand All @@ -156,6 +155,55 @@ Ordinary mode selects every workflow E2E except `Exact staging Brev Launchable`.
Launchable mode runs only `Exact staging Brev Launchable`.
Full mode selects every workflow E2E, including `Exact staging Brev Launchable`, in the same workflow run.

## Inspect the Edition's Main-Push Runs

For overnight diagnosis, start from the trusted frozen plan. Every push to `main` already starts its own immutable run selecting every workflow E2E, so do not dispatch a duplicate full run merely to begin the overnight loop:

```bash
export PLAN_PATH=/path/to/downloaded-release-edition-plan/plan.json
FROZEN_CANDIDATE_SHA="$(node -p "require(process.env.PLAN_PATH).candidateCommit")"
export EDITION_CUTOFF_AT="$(node -p "require(process.env.PLAN_PATH).authorization.cutoffAt")"
CUTOFF_SECONDS="$(node -p "Date.parse(process.env.EDITION_CUTOFF_AT) / 1000")"
WINDOW_START_SECONDS="$((CUTOFF_SECONDS - 8 * 60 * 60))"
RUNS="$(gh run list --repo NVIDIA/NemoClaw --workflow e2e.yaml \
--event push --branch main --limit 200 \
--json databaseId,createdAt,headSha,status,conclusion,url)"
MATCHES="$(jq -c --argjson start "$WINDOW_START_SECONDS" --argjson end "$CUTOFF_SECONDS" \
'[.[] | select((.createdAt | fromdateiso8601) >= $start and
(.createdAt | fromdateiso8601) <= $end)] | sort_by(.createdAt)' <<<"$RUNS")"
jq -e 'length >= 1' <<<"$MATCHES" >/dev/null
jq -e 'all(.[]; (.headSha // "") | test("^[0-9a-f]{40}$"))' <<<"$MATCHES" >/dev/null
```
Comment thread
coderabbitai[bot] marked this conversation as resolved.

The inventory covers the 8:00 AM–4:00 PM merge window ending at the plan's exact cutoff. More than one run is expected. Fetch `origin/main` and require every selected `headSha` to be an ancestor of `FROZEN_CANDIDATE_SHA`; exclude and report any unrelated run instead of silently treating it as edition evidence. If a merge's push run is not visible yet, report it as pending and poll. Do not silently substitute full mode.

Each run is keyed to its own `headSha`; a later push does not cancel an earlier main-push run. `.github/workflows/e2e-main-retry.yaml` may rerun failed jobs from a non-superseded main-push run up to two times. Keep the source run ID, attempt, SHA, job conclusion, artifacts, and retry evidence together. The tag remains bound only to the frozen plan.

Watch and inspect each selected run even when it fails:

```bash
gh run watch "<run-id>" --repo NVIDIA/NemoClaw
gh run view "<run-id>" --repo NVIDIA/NemoClaw \
--json status,conclusion,headSha,jobs,url
gh run view "<run-id>" --repo NVIDIA/NemoClaw --log-failed
```

Classify failures before choosing selective reruns. Dispatch full mode only for an explicit full or release-candidate rerun request.

## Operate the Overnight Loop

For the frozen edition, keep one agent session active from 4:00 PM through the 8:00 AM handoff. Do not start competing agents over the same failure set. Repeat this sequence until the handoff boundary:

1. inspect newly completed main-push, automatic-retry, or selective E2E jobs and exact-SHA post-merge advisor findings;
2. choose the highest-impact unresolved failure that is not already owned by a prepared fix;
3. classify it as a product regression, flaky test, infrastructure failure, or stale test;
4. prepare the smallest focused fix or justified test cleanup and run its deterministic checks;
5. open or update a fix PR without merging it during the freeze;
6. dispatch only the selective rerun needed to test the diagnosis; and
7. update the shared handoff state, then immediately choose the next actionable item.

Do not wait idly for an unrelated rerun when another unresolved failure can be diagnosed. Continue across the 4:00 AM tag without changing the frozen candidate or treating the tag as E2E success. At 8:00 AM, stop the loop and hand over every unresolved failure, rerun, and prepared PR. If the active agent cannot continue before then, transfer the same state to one replacement agent.

## Resolve the Candidate

Run from a trusted NemoClaw checkout:
Expand All @@ -166,9 +214,7 @@ git fetch --prune origin main
CANDIDATE_SHA="$(git rev-parse origin/main)"
```

For a pre-tag request, use the full candidate SHA from the generated release plan.
Require that SHA to equal `origin/main` before dispatch.
Stop and regenerate the release plan when they differ.
For a frozen-edition request, read `FROZEN_CANDIDATE_SHA` from the generated release plan and compare it with `CANDIDATE_SHA`, which is the current trusted `origin/main` dispatch ref. When they match, the rerun is also bound to the frozen candidate. If `main` advanced after the cutoff, keep the plan frozen and describe new dispatches as current-main or next-edition validation; the current direct-main workflow does not dispatch an older ancestor. Do not use manual PR checkout inputs to bypass that boundary.

Record `CANDIDATE_SHA` for every dispatch.
Do not use a relative revision in the evidence report.
Expand Down Expand Up @@ -276,15 +322,15 @@ gh api "repos/NVIDIA/NemoClaw/actions/runs/$RUN_ID/jobs?filter=latest&per_page=1
>"$EVIDENCE_DIR/jobs-latest-$RUN_ID.json"
```

For full-mode or release evidence, collect every attempt for the matrix-preserving ledger:
For full mode, collect every attempt so the validator can retain successful evidence from an earlier attempt of the same workflow run:

```bash
gh api --paginate --slurp \
"repos/NVIDIA/NemoClaw/actions/runs/$RUN_ID/jobs?filter=all&per_page=100" \
>"$EVIDENCE_DIR/jobs-$RUN_ID.json"
```

Reuse `run-$RUN_ID.json` and `jobs-$RUN_ID.json` as the `nemoclaw-maintainer-cut-release-tag` manifest inputs and as the full-mode validator inputs. Do not fetch the same run again. `jobs-latest-$RUN_ID.json` is only for ordinary and Launchable modes.
Use `jobs-$RUN_ID.json` as the full-mode validator input. `jobs-latest-$RUN_ID.json` is only for ordinary and Launchable modes.

For ordinary and Launchable modes, require `run-$RUN_ID.json` to report:

Expand Down Expand Up @@ -332,12 +378,12 @@ The validator requires:
- `cleanup.json` to report the same workspace as `ABSENT`.

A skipped, cancelled, queued, or failed Launchable E2E job is not evidence.
A Launchable-mode run is not full-mode or pre-tag release evidence.
A Launchable-mode run is not full-mode evidence.
A missing, mismatched, or failed cleanup receipt is not evidence.

## Bind Release Evidence
## Bind Advisory Evidence

If no release plan exists, label a successful full run against `origin/main` as provisional release evidence.
If no release plan exists, label a successful full run against `origin/main` as advisory E2E evidence.
Return:

- candidate SHA;
Expand All @@ -347,13 +393,9 @@ Return:
- Launchable E2E identity; and
- cleanup result.

If the release candidate SHA changes, discard the earlier full run and dispatch full mode for the new SHA.
No release-note-only delta exception is currently defined.

When `nemoclaw-maintainer-cut-release-tag` invokes this skill, return the validated fields for its pre-tag E2E evidence ledger.
The trusted `dispatch.json` receipt proves that full mode used empty selectors and included `Exact staging Brev Launchable`.
The release evidence ledger proves the result of each workflow E2E.
Do not ask for the release confirmation phrase in this skill.
Bind every result to the tested SHA. For the frozen edition, classify failures and prepare fixes for the next merge window.
Return the trusted dispatch, test, Launchable, and cleanup receipts needed for diagnosis.
Never treat success as tag authorization or failure as a reason to delay the 4 AM tag.

## Access Failures

Expand Down
Loading
Loading