From a0e189deb3a6dcb74182c7e968c5f97e8c646171 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas Date: Tue, 5 May 2026 14:29:35 -0700 Subject: [PATCH 01/40] feat(skills): add nemoclaw-maintainer-verify-stale MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a maintainer skill that automates verifying whether old bug reports still reproduce against the latest NemoClaw release. The skill picks candidate issues opened against older versions, reuses or provisions a Brev Linux box (CPU or GPU based on the bug profile), runs the extracted reproducer, scores confidence, and posts an evidence-backed comment with either a `fixed-on-latest` or `verify-inconclusive` label. Tag-only — never auto-closes. v1 scope is intentionally narrow: Linux only, no third-party integration credentials, no service-account bot. Each maintainer runs it under their own gh credentials. Companion changes: - Adds the new skill to the maintainer table in `nemoclaw-skills-guide` (count 7 -> 8 maintainer skills, 18 -> 19 total). - Adds Step 7 to `nemoclaw-maintainer-cut-release-tag` that sweeps `fixed-on-latest` and `verify-inconclusive` labels off all open issues at release time, so the next verify-stale run re-evaluates against the new latest. Verification records remain in comment history; only the labels are reset. Signed-off-by: Prekshi Vyas --- .../SKILL.md | 14 + .../nemoclaw-maintainer-verify-stale/SKILL.md | 354 ++++++++++++++++++ .agents/skills/nemoclaw-skills-guide/SKILL.md | 7 +- 3 files changed, 372 insertions(+), 3 deletions(-) create mode 100644 .agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md diff --git a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md index b33ae020da3..aae53b1a806 100644 --- a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md @@ -127,6 +127,20 @@ git ls-remote --tags origin | grep -E '(|latest)' Confirm both tags point to the same commit on the remote. +## Step 7: Sweep Stale-Issue Verification Labels + +Strip `fixed-on-latest` and `verify-inconclusive` from all open issues so the next `nemoclaw-maintainer-verify-stale` run re-evaluates against the new release. Without this sweep, "latest" drifts and verifications go silently stale. + +```bash +for label in fixed-on-latest verify-inconclusive; do + gh issue list --repo NVIDIA/NemoClaw --state open --label "$label" \ + --json number -q '.[].number' \ + | xargs -I{} gh issue edit {} --repo NVIDIA/NemoClaw --remove-label "$label" +done +``` + +The verification record itself stays in each issue's comment history — only the labels are reset. + ## Important Notes - NEVER tag without explicit user confirmation of the version. diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md new file mode 100644 index 00000000000..546883105d1 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -0,0 +1,354 @@ +--- +name: nemoclaw-maintainer-verify-stale +description: Verify whether old NVIDIA/NemoClaw bug reports still reproduce against the latest release. Picks candidate issues opened against older versions, reuses or provisions a Brev Linux box (CPU or GPU), attempts reproduction, scores confidence, and posts an evidence-backed comment with a label (fixed-on-latest or verify-inconclusive). Tag-only — never auto-closes. Linux-only in v1; Windows, macOS, and integration-token-dependent issues are skipped. Trigger keywords - verify stale, verify fixed, reproduce on latest, stale issue, old bug, fixed-on-latest, verify-inconclusive, drain backlog, brev verify. +user_invocable: true +--- + + + + +# NemoClaw Maintainer — Verify Stale Issues + +Automates the manual loop of "spin up a Brev box, install latest NemoClaw, try to reproduce an old bug, comment with findings." Drains the bug backlog by surfacing issues that have been silently fixed. + +This skill is the outbound counterpart to `nemoclaw-diagnosis` (which files issues from CI failures). Diagnosis fills the queue; this drains it. + +--- + +## Step 1: Determine Mode + +**Single-issue mode** — user provides an issue number: + +```bash +gh issue view --repo NVIDIA/NemoClaw \ + --json number,title,body,labels,url,author,createdAt,comments +``` + +**Batch mode** — user says "batch", "weekly", or provides no number. Cap at 20 issues per run. + +```bash +gh issue list --repo NVIDIA/NemoClaw --state open --limit 100 \ + --label bug \ + --json number,title,body,labels,url,author,createdAt,comments +``` + +In batch mode, work through items one at a time. Present each verification plan and wait for approval before any Brev provisioning. + +--- + +## Step 2: Detect the Latest NemoClaw Version + +```bash +LATEST=$(gh release view --repo NVIDIA/NemoClaw --json tagName -q .tagName) +echo "Latest release: $LATEST" +``` + +This is the version the skill will verify against. Record it — every comment must cite it. + +--- + +## Step 3: Filter Candidates + +Apply these rules in order. Drop any issue that fails a rule. + +**Issue-type allowlist:** must have `bug` label. +**Issue-type skip:** drop if any of `enhancement`, `documentation`, `wontfix`, `needs-info`, `security`. + +**Platform skip (Linux-only in v1):** drop if any of `Platform: Windows/WSL`, `Platform: MacOS`, `Platform: macOS`. Keep `Platform: Ubuntu`, `Platform: DGX Spark`, `Platform: GB10`, `Platform: All`, or no platform label. + +**Integration skip (deferred to v2):** drop if any of `Integration: Slack`, `Integration: Discord`, `Integration: Telegram`, `Integration: Hermes`, `Integration: OpenClaw`, `Integration: WeChat`. These need third-party credentials a fresh Brev box cannot provide. + +**Component allowlist (must have at least one):** `NemoClaw CLI`, `Sandbox`, `OpenShell`, `Docker`, `Getting Started`, or any `Platform:` label that survived the platform skip. + +**Idempotency:** drop if any comment body on the issue contains ``. The skill never re-verifies an issue within the same release window. (The release sweep in `nemoclaw-maintainer-cut-release-tag` clears prior `fixed-on-latest` and `verify-inconclusive` labels on each release, which is what re-opens the candidate set.) + +**Candidate rule:** keep the issue if **either**: + +- The reported version (parsed from body or labels — see Step 4) is **2 or more minor versions behind** `$LATEST`, **or** +- The issue is **older than 7 days** AND a specific version is parseable from its body or labels. + +--- + +## Step 4: Parse Reported Version + +Search the body and labels for `v?0\.0\.\d+`. Sources, in order of trust: + +1. A label matching the version regex (e.g. `v0.0.32`). +2. The first version match in the issue body. +3. Comments by the original reporter. + +If no version can be parsed, drop the issue from the candidate set — we cannot establish "previous version". + +--- + +## Step 5: Classify the Verification Environment + +**CPU vs GPU:** GPU if any of these signals are present, else CPU. + +- Labels: `Platform: GB10`, `Platform: DGX Spark`. +- Body keywords: `cuda`, `nvidia-smi`, `inference`, `model serving`, `H100`, `A100`, `GB10`, `DGX`. + +CPU default keeps cost low. Only escalate to GPU when the reproducer needs one. + +--- + +## Step 6: Extract the Reproducer + +Try in order, stop at the first that works: + +1. **Verbatim extraction:** the first fenced code block in the issue body that contains a `nemoclaw` invocation. No confidence penalty. +2. **LLM synthesis:** if no fenced block matches, synthesize a shell script from the narrative bug report. Apply a **−30 confidence penalty** later. +3. **Give up:** if neither produces a runnable script, mark the issue `verify-inconclusive`, post a short comment explaining why, and move on. Do not provision a Brev box. + +Save the chosen script to `./reproducer.sh`. Both verbatim and synthesized scripts will be quoted in the final comment as evidence. + +--- + +## Step 7: Reuse or Provision a Brev Box + +The skill prefers reuse over provisioning. A pool of `verify-stale-*` boxes (CPU and GPU) can be kept warm; reuse the matching one if available, otherwise provision. + +```bash +# Ensure an active Brev session. brev ls fails if not authenticated. +brev ls --json >/dev/null 2>&1 || brev login + +# Determine class from Step 5: "cpu" or "gpu" +INSTANCE_CLASS="cpu" # or "gpu" + +INSTANCES=$(brev ls --json) + +# Look for an existing running verify-stale-* box matching the required class. +# CPU boxes have no .gpu field set; GPU boxes do. +EXISTING=$(echo "$INSTANCES" | jq -r --arg class "$INSTANCE_CLASS" ' + .[]? + | select(.name | startswith("verify-stale-")) + | select(.status == "RUNNING") + | select(($class == "gpu" and (.gpu // "" != "")) + or ($class == "cpu" and (.gpu // "" == ""))) + | .name' | head -1) + +PROVISIONED_NEW=0 + +if [ -n "$EXISTING" ]; then + INSTANCE_NAME="$EXISTING" + echo "Reusing existing verification box: $INSTANCE_NAME" +else + # Concurrency cap: refuse if 4+ verify-stale-* boxes are already running. + RUNNING=$(echo "$INSTANCES" | jq '[.[]? | select(.name | startswith("verify-stale-"))] | length') + if [ "$RUNNING" -ge 4 ]; then + echo "ERROR: 4 verify-stale boxes already running. Wait for one to finish or reuse." + exit 1 + fi + + INSTANCE_NAME="verify-stale-${ISSUE_NUMBER}-$(date +%s)" + + if [ "$INSTANCE_CLASS" = "gpu" ]; then + # brev create auto-selects the cheapest GPU meeting the defaults + # (>=20GB VRAM, >=500GB disk, compute >=8.0). Override with --type if needed. + brev create "$INSTANCE_NAME" + else + # CPU case: pass an explicit --type from your team's allowed CPU SKUs + # (brev create defaults to GPU). Pin this in your team config. + brev create "$INSTANCE_NAME" --type "" + fi + + PROVISIONED_NEW=1 +fi + +# Cleanup runs on success, error, and SIGINT. +# Delete only what we provisioned. Reused boxes stay warm for next time. +trap '[ "$PROVISIONED_NEW" = "1" ] && brev delete "$INSTANCE_NAME" --yes || true' EXIT +``` + +Wallclock cap per verification: **15 minutes** including reuse-check, install, and reproduction. If a provisioned box isn't ready in time, abort and treat as an infra failure (Step 11). + +--- + +## Step 8: Reset, Install Latest, Run the Reproducer + +Even on a reused box, reset NemoClaw state before installing — hermeticity matters more than the few seconds saved. + +```bash +# Reset prior NemoClaw state on the box (safe no-op on a fresh box). +brev exec "$INSTANCE_NAME" "rm -rf ~/.nemoclaw 2>/dev/null; sudo rm -f /usr/local/bin/nemoclaw 2>/dev/null; true" + +# Install latest NemoClaw. +brev exec "$INSTANCE_NAME" "curl -fsSL https://nemoclaw.nvidia.com/install.sh | bash" +brev exec "$INSTANCE_NAME" "nemoclaw --version" + +# Copy and run the extracted reproducer; capture full transcript. +brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh +brev exec "$INSTANCE_NAME" @reproducer-runner.sh 2>&1 | tee ./transcript.log +``` + +If the install itself fails (e.g. installer regression — see #3058 for a current example), this is an **infra failure** — see Step 11. Do not score or label the issue. + +For interactive debugging when something looks off: + +```bash +brev shell "$INSTANCE_NAME" +``` + +--- + +## Step 9: Score Confidence + +Start at 0. Apply each rule that fires. + +| Signal | Delta | +|---|---| +| Reproducer ran cleanly on latest, exit 0, expected output observed | +50 | +| Commits between reported version and `$LATEST` touch the implicated component (`git log v..$LATEST -- `) | +25 | +| A merged PR mentions this issue number or its symptom | +25 | +| Reproducer was LLM-synthesized, not extracted verbatim | −30 | +| Any partial error, warning, or flaky behavior in the repro run | −50 | + +Total is clamped to `[0, 100]`. + +**Action:** + +| Score | Label | Comment | +|---|---|---| +| ≥85 | `fixed-on-latest` | Evidence-rich, no @-mention. | +| 60–84 | `fixed-on-latest` | Evidence-rich, **@-mention the original reporter** to confirm. | +| <60 | `verify-inconclusive` | Short, honest "couldn't verify" explanation. | + +The skill **never closes issues**. A maintainer pulls that trigger after reviewing the label and comment. + +--- + +## Step 10: Compose and Post the Comment + +**Redaction pass before posting.** Strip from any text quoted out of the issue body: + +- Anything matching `(?i)(token|secret|password|api[_-]?key|bearer)[^\n]*[:=][^\n]*` +- URLs containing `@` (basic-auth credentials). +- File paths under the reporter's home directory (replace with `~/`). + +**Comment template:** + +````markdown +## Stale-issue verification — automated + +**Reported on:** v0.0.31 +**Verified on:** v0.0.35 (commit abc1234) +**Environment:** Brev () / Ubuntu 22.04 / +**Reproducer source:** extracted verbatim from issue body | LLM-synthesized from narrative + +**Result:** not reproducible — exit 0, expected output observed. +**Confidence:** 88 / 100. Labelling `fixed-on-latest`. + +
Reproduction transcript + +```text + +``` + +
+ +
Relevant changes since v0.0.31 + +- abc1234 — fix: +- def5678 — refactor: + +
+ +If this verification is wrong, please reopen the issue with a comment and the skill will re-verify on the next release. + + +```` + +The trailing HTML comment is the **idempotency marker** Step 3 looks for. Never omit it. + +**Post the comment and apply the label:** + +```bash +gh issue comment "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --body-file comment.md +gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "fixed-on-latest" +# or for <60: +# gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "verify-inconclusive" +``` + +--- + +## Step 11: Infra Failure Handling + +If reuse-check, provisioning, install, or the test harness itself fails (not the reproducer): + +- Print the error. +- Apply **no label** — infra failures must not pollute the verification record. +- Post a short comment **only if explicitly requested by the invoking user**. Default is silent move-on. +- Continue to the next candidate in batch mode. + +The next weekly run retries naturally. + +--- + +## Step 12: Log to Activity + +After each issue (verified, inconclusive, or infra-failed), append to `~/development/daily-rhythm/activity/nemoclaw-verify-stale-log.md`. + +```markdown +### NVIDIA/NemoClaw# — +**Date:** YYYY-MM-DD +**Reported on:** v0.0.31 +**Verified on:** v0.0.35 +**Environment:** CPU | GPU (<instance type>) +**Box:** reused <name> | provisioned <name> +**Reproducer:** verbatim | synthesized | none +**Confidence:** 88 / 100 +**Label applied:** fixed-on-latest | verify-inconclusive | none (infra) +**Brev wall time (approx):** N min + +--- +``` + +Create the file if missing, with this header: + +```markdown +# NemoClaw — Verify Stale Log + +A running record of stale-issue verification runs on NVIDIA/NemoClaw. +Persisted via daily-rhythm to GitLab. + +--- +``` + +At end of a batch session, prepend a session summary: + +```markdown +## YYYY-MM-DD — Verify Session +**Issues considered:** N +**Verified `fixed-on-latest`:** N +**Marked `verify-inconclusive`:** N +**Skipped (Windows / macOS / integration / no version):** N +**Infra failures:** N +**Brev wall time:** N min · approx $X.XX + +--- +``` + +Never stage or commit the log to the NemoClaw repo. + +--- + +## Cadence + +- **Weekly cron** — Monday morning, batch mode, ≤20 issues. +- **Manual** — invoke with a single issue number anytime. + +--- + +## Out of Scope (v1) + +- Auto-closing issues. Always tag-only; a human pulls the trigger. +- macOS verification. Brev offers no macOS instances and local-laptop runs are not unattended. +- Issues requiring third-party integration credentials (Slack, Discord, Telegram, Hermes, OpenClaw, WeChat). +- Service-account bot identity. v1 runs under each maintainer's own GitHub credentials. +- Versioned labels. A single `fixed-on-latest` label is swept on each release cut. + +--- + +## Companion Behavior + +`nemoclaw-maintainer-cut-release-tag` sweeps `fixed-on-latest` and `verify-inconclusive` from all open issues at release time. Without that sweep, "latest" drifts and verifications go stale silently. diff --git a/.agents/skills/nemoclaw-skills-guide/SKILL.md b/.agents/skills/nemoclaw-skills-guide/SKILL.md index a59145045aa..7deac52b521 100644 --- a/.agents/skills/nemoclaw-skills-guide/SKILL.md +++ b/.agents/skills/nemoclaw-skills-guide/SKILL.md @@ -21,10 +21,10 @@ The prefix in each skill name indicates who it is for. For end users operating a NemoClaw sandbox. Covers installation, inference configuration, network policy management, monitoring, remote deployment, security configuration, workspace management, and reference material. -### `nemoclaw-maintainer-*` (7 skills) +### `nemoclaw-maintainer-*` (8 skills) For project maintainers. -Covers the daily maintainer cadence (morning standup, daytime loop, evening handoff), cutting releases, finding PRs to review, normalizing issue and PR title tags, and performing security code reviews. +Covers the daily maintainer cadence (morning standup, daytime loop, evening handoff), cutting releases, finding PRs to review, normalizing issue and PR title tags, performing security code reviews, and verifying whether stale bug reports still reproduce on the latest release. ### `nemoclaw-contributor-*` (2 skills) @@ -60,6 +60,7 @@ Covers creating pull requests that follow the project template and drafting docu | `nemoclaw-maintainer-find-review-pr` | Find open PRs labeled security + priority-high, link each to its issue, detect duplicates, and present a review summary. | | `nemoclaw-maintainer-normalize-title-tags` | Preview and remove bracketed `NemoClaw` title tags from issues and PRs case-insensitively, even when the tag appears later in the title. | | `nemoclaw-maintainer-security-code-review` | Perform a 9-category security review of a PR or issue, producing per-category PASS/WARNING/FAIL verdicts. | +| `nemoclaw-maintainer-verify-stale` | Verify whether old bug reports still reproduce on latest. Reuses or provisions a Brev box (CPU or GPU), runs the extracted reproducer, scores confidence, and posts an evidence-backed comment with `fixed-on-latest` or `verify-inconclusive`. Tag-only — never auto-closes. | ### Contributor Skills @@ -82,6 +83,6 @@ Skills are cumulative. Each role includes the skills from the roles above it: |------|----------------|-------|------------| | User | `nemoclaw-user-*` | 9 | `nemoclaw-user-get-started` | | Contributor | `nemoclaw-user-*` + `nemoclaw-contributor-*` | 11 | `nemoclaw-user-overview` | -| Maintainer | All skills | 18 | `nemoclaw-maintainer-morning` | +| Maintainer | All skills | 19 | `nemoclaw-maintainer-morning` | After identifying the role, present the applicable skills from the Skill Catalog above and recommend the starting skill. From 1ea1559d46485f81b1bc4360180c0fef75bda601 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Tue, 5 May 2026 14:42:32 -0700 Subject: [PATCH 02/40] fix(verify-stale): tighten version detection and parsing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three flaws surfaced when dry-running the Step 3 candidate filter against the live open-issue list (141 open bugs). 1. Step 2 used `gh release view` to detect "latest". NemoClaw tags but does not publish GitHub releases, so this returned empty. Added a fallback that picks the highest semver from `git ls-remote --tags`, which is the load-bearing path today. 2. Step 3 said "2 or more minor versions behind". Wrong vocabulary for `0.0.x` repos — patch is what's iterating. Reworded to "at least 2 versions behind in the rightmost-incrementing component", with a concrete `0.0.x` example and a forward-compat note for when NemoClaw moves to `0.1.x`. 3. Step 4 specified a naive `v?0\.0\.\d+` regex that matched IP-like strings in bug bodies (Ollama bind `0.0.0.0:11434`, loopback `127.0.0.1`, etc.) and produced phantom v0.0.0 / v0.0.1 candidates. Tightened to require `v` prefix + word boundaries first, with a context-anchored fallback that only matches `0.0.X` on lines containing `nemoclaw` or `version`. Added a clamp that rejects any parsed version greater than `$LATEST` (catches roadmap labels like `v0.0.35` from being treated as "reported on"). After these fixes the dry-run produces 33 plausible candidates out of 141 open bugs, with credible reported-version assignments verified against issue bodies. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 546883105d1..b414f670605 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -38,8 +38,18 @@ In batch mode, work through items one at a time. Present each verification plan ## Step 2: Detect the Latest NemoClaw Version +Try GitHub releases first; fall back to the highest semver git tag if no release is published. NemoClaw currently tags but does not publish releases, so the fallback is the load-bearing path today. + ```bash -LATEST=$(gh release view --repo NVIDIA/NemoClaw --json tagName -q .tagName) +LATEST=$(gh release view --repo NVIDIA/NemoClaw --json tagName -q .tagName 2>/dev/null) + +if [ -z "$LATEST" ]; then + LATEST=$(git ls-remote --tags --refs git@github.com:NVIDIA/NemoClaw.git \ + | awk -F/ '{print $NF}' \ + | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' \ + | sort -V | tail -1) +fi + echo "Latest release: $LATEST" ``` @@ -64,18 +74,22 @@ Apply these rules in order. Drop any issue that fails a rule. **Candidate rule:** keep the issue if **either**: -- The reported version (parsed from body or labels — see Step 4) is **2 or more minor versions behind** `$LATEST`, **or** +- The reported version (parsed from body or labels — see Step 4) is **at least 2 versions behind** `$LATEST` in the rightmost-incrementing component, **or** - The issue is **older than 7 days** AND a specific version is parseable from its body or labels. +For NemoClaw's current `0.0.x` line, "rightmost-incrementing component" is the patch number — a v0.0.31 report against a v0.0.34 latest is 3 versions behind. Once NemoClaw moves to `0.1.x` or higher, the rule applies to the next-rightmost component instead. Pick whichever component is actively iterating. + --- ## Step 4: Parse Reported Version -Search the body and labels for `v?0\.0\.\d+`. Sources, in order of trust: +Sources, in order of trust: + +1. A label that exactly matches a released version pattern (e.g. `v0.0.32`). Reject labels that match a version newer than `$LATEST` — those are roadmap/release-target labels, not "reported on". +2. The body, with a tight regex: first try `\bv0\.0\.(\d+)\b` (require the `v` prefix and word boundaries). Only if nothing matches, fall back to `\b0\.0\.(\d+)\b` **but only on lines containing `nemoclaw` or `version` (case-insensitive)**. The loose form alone is unsafe — without context anchoring, it matches `0.0.0.0:11434` (Ollama bind address), `127.0.0.1`, IPs in log lines, etc., and produces phantom "v0.0.0" or "v0.0.1" candidates. +3. Comments by the original reporter (same regex as the body). -1. A label matching the version regex (e.g. `v0.0.32`). -2. The first version match in the issue body. -3. Comments by the original reporter. +After parsing, **clamp**: if the parsed version is greater than `$LATEST`, treat it as unparseable. This catches roadmap labels that slipped past step 1. If no version can be parsed, drop the issue from the candidate set — we cannot establish "previous version". From 6996e2923d28351c08f3694807a95792e5174271 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Tue, 5 May 2026 14:51:16 -0700 Subject: [PATCH 03/40] fix(verify-stale): generalize version regex and validate against tags MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two more issues surfaced from a deeper second-pass dry-run. 1. The Step 4 regex was hardcoded to v0.0.x. NemoClaw is on that line today, but the skill should keep working when it moves to v0.1.x or higher. Generalized to `\bv\d+\.\d+\.\d+\b` with a context-anchored `\b\d+\.\d+\.\d+\b` fallback on lines containing nemoclaw or version (so the loose form doesn't match `0.0.0.0:11434` and `127.0.0.1`). 2. Some bug reports cite NemoClaw versions that were never tagged (e.g. `v0.1.0` × 3 issues, calver `2026.3.11` × 1). Without a tag existence check the skill would point Brev verification at a version that does not exist. Added a `git ls-remote --tags` validation pass that drops the version if no matching tag is found. Also added an implementer note documenting a real failure mode caught during the dry-run: a naive `[scan(primary)] | first | .[0] | tonumber // [scan(fallback)] | ...` pipeline silently dropped 9 valid candidates because `null | first` errors when scan returns empty, and the `//` chain did not propagate cleanly to the fallback. The SKILL.md regex itself was correct — the failure was implementation-level fragility in how the two passes were composed. The note tells implementers to bind each pass to a named variable, coalesce at the end, and explicitly test the empty-match path. After these changes the live dry-run produces 47 plausible candidates (out of 141 open bugs) and the 4 residual drops are all reporter typos that cite a version that was never released — correctly unverifiable. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 38 ++++++++++++++++--- 1 file changed, 33 insertions(+), 5 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index b414f670605..93d8930eecd 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -83,15 +83,43 @@ For NemoClaw's current `0.0.x` line, "rightmost-incrementing component" is the p ## Step 4: Parse Reported Version +The regex is intentionally **release-line agnostic**. Today NemoClaw ships `v0.0.x`, but the same parser must keep working when it moves to `v0.1.x`, `v1.x.x`, or anything else. Don't hardcode the major/minor digits. + Sources, in order of trust: -1. A label that exactly matches a released version pattern (e.g. `v0.0.32`). Reject labels that match a version newer than `$LATEST` — those are roadmap/release-target labels, not "reported on". -2. The body, with a tight regex: first try `\bv0\.0\.(\d+)\b` (require the `v` prefix and word boundaries). Only if nothing matches, fall back to `\b0\.0\.(\d+)\b` **but only on lines containing `nemoclaw` or `version` (case-insensitive)**. The loose form alone is unsafe — without context anchoring, it matches `0.0.0.0:11434` (Ollama bind address), `127.0.0.1`, IPs in log lines, etc., and produces phantom "v0.0.0" or "v0.0.1" candidates. -3. Comments by the original reporter (same regex as the body). +1. A label that exactly matches a real released version (e.g. `v0.0.32`). Reject labels that match a version newer than `$LATEST` — those are roadmap/release-target labels, not "reported on". +2. The body. Two-pass regex: + - **Primary:** `\bv\d+\.\d+\.\d+\b` — require the `v` prefix and word boundaries. Matches any `vMAJOR.MINOR.PATCH`. + - **Fallback:** `\b\d+\.\d+\.\d+\b` **only on lines containing `nemoclaw` or `version` (case-insensitive)**. Without that line filter, the fallback alone matches IPs and bind addresses (`0.0.0.0:11434`, `127.0.0.1`) and other unrelated semver-ish strings, producing phantom candidates. +3. Comments by the original reporter (same two-pass regex as the body). + +After parsing, run two validation passes: + +- **Clamp future:** if the parsed version is greater than `$LATEST`, treat it as unparseable. This catches roadmap labels that slipped past source 1. +- **Validate against tags:** confirm the parsed version exists as an actual git tag. This catches reporter typos such as `NemoClaw: v0.1.0` (no such release in the current `v0.0.x` line) and calver mistakes like `NemoClaw: 2026.3.11` (date string, not a tag). + +```bash +git ls-remote --tags --refs git@github.com:NVIDIA/NemoClaw.git \ + | awk -F/ '{print $NF}' \ + | grep -Fx "v$PARSED_VERSION" >/dev/null \ + || PARSED_VERSION="" # treat as unparseable +``` + +If no version survives both passes, drop the issue from the candidate set — we cannot establish "previous version". + +### Implementer note: regex-pipeline pitfall + +In the v1 dry-run, a naive jq pipeline that chained the primary and fallback regexes via `[scan(primary)] | first | .[0] | tonumber // [scan(fallback)] | first | .[0] | tonumber` silently dropped 9 real candidates (e.g. #2861 with `NemoClaw 0.0.32` in body, #2604 with `NemoClaw: 0.0.28`). When the primary regex matched empty, `null | first` errored, and `//` did not propagate cleanly to the fallback. -After parsing, **clamp**: if the parsed version is greater than `$LATEST`, treat it as unparseable. This catches roadmap labels that slipped past step 1. +Whichever language you implement in, structure the parser so the empty-match path returns null cleanly (not an error). Bind each pass to a named variable and `coalesce` them at the end: + +```text +primary := first match of \bv\d+\.\d+\.\d+\b in body (or null) +fallback := first match of \b\d+\.\d+\.\d+\b on nemoclaw/version lines (or null) +result := primary ?? fallback +``` -If no version can be parsed, drop the issue from the candidate set — we cannot establish "previous version". +Always test against an issue body with **no** version mention before trusting the result — that's the path that exercises the empty-match handling. --- From 4129edcdc2b7cfabcdf0cca7919cff186c3ed9e7 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Tue, 5 May 2026 15:00:00 -0700 Subject: [PATCH 04/40] fix(verify-stale): anchor version regex to nemoclaw proximity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third dry-run round surfaced a precision issue. The previous regex matched any `\bv\d+\.\d+\.\d+\b` anywhere in the body, then validated against the tag list. That picked up versions belonging to *other* products that share the v0.0.x line — most often OpenShell, but also Node.js (v22.x.x), and other dependencies. Tag validation then correctly rejected the non-NemoClaw ones, but on bodies where multiple products were listed (e.g. `Environment: openshell 0.0.4, nemoclaw 0.1.0, Node.js v22.16.0`) the parser would land on OpenShell's tag-valid v0.0.4 and report it as the NemoClaw version, producing a false-positive candidate. 12 such collisions exist in the current backlog. Fix: require the version to follow `nemoclaw` (case-insensitive) within 80 non-letter, non-newline characters. The anchored regex `(?i)nemoclaw[^a-z\n]{0,80}v?(\d+\.\d+\.\d+)` matches `NemoClaw v0.0.32`, `nemoclaw 0.0.28`, `- NemoClaw: v0.0.16`, but not `openshell 0.0.4` followed later by `nemoclaw 0.1.0`. Combined with the existing tag-list validation, this collapses the four error classes — typos, calver dates, roadmap labels, and product collisions — into one "version must be a real NemoClaw release tag, mentioned next to NemoClaw" check. Also expanded the implementer note to cover three concrete failure modes hit during the dry-run: - Empty-match handling (`[]` flowing into `first | .[0]` errors). - Capture-group inconsistency between branches in a parser pipeline. - Variable-scoping bug in `select($tags | index(.))` where `.` rebinds to `$tags` and silently passes invalid labels. After this change the live dry-run produces 33 candidates out of 141 open bugs — same count as the very first round, but every candidate's parsed version is now anchored to NemoClaw and validated against the real tag list. The earlier wider counts (47, 48) included roughly 12-15 issues that would have been wasted Brev runs. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 50 +++++++++++-------- 1 file changed, 29 insertions(+), 21 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 93d8930eecd..aa3bee9a2a5 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -87,39 +87,47 @@ The regex is intentionally **release-line agnostic**. Today NemoClaw ships `v0.0 Sources, in order of trust: -1. A label that exactly matches a real released version (e.g. `v0.0.32`). Reject labels that match a version newer than `$LATEST` — those are roadmap/release-target labels, not "reported on". -2. The body. Two-pass regex: - - **Primary:** `\bv\d+\.\d+\.\d+\b` — require the `v` prefix and word boundaries. Matches any `vMAJOR.MINOR.PATCH`. - - **Fallback:** `\b\d+\.\d+\.\d+\b` **only on lines containing `nemoclaw` or `version` (case-insensitive)**. Without that line filter, the fallback alone matches IPs and bind addresses (`0.0.0.0:11434`, `127.0.0.1`) and other unrelated semver-ish strings, producing phantom candidates. -3. Comments by the original reporter (same two-pass regex as the body). +1. **Labels.** Any label that exactly matches `^v\d+\.\d+\.\d+$` AND appears in the repo's tag list. Labels matching the regex but absent from tags (e.g. `v0.0.35` as a *release-target* milestone before that version ships) are roadmap markers, not "reported on" — drop them. +2. **Body.** Use a **proximity-anchored** regex: `(?i)nemoclaw[^a-z\n]{0,80}v?(\d+\.\d+\.\d+)`. This matches a version that follows `nemoclaw` within 80 non-letter, non-newline characters, capturing just the semver. The anchoring is load-bearing — without it the parser also picks up `openshell 0.0.4`, Node.js `v22.16.0`, IP addresses (`0.0.0.0:11434`, `127.0.0.1`), and other near-NemoClaw products that happen to share the `v0.0.x` line. (This was confirmed in the dry-run: a non-anchored parser produced 12 false-positive candidates whose smallest tag-valid version was actually OpenShell's, not NemoClaw's.) +3. **Comments by the original reporter** — same anchored regex as the body. -After parsing, run two validation passes: +Collect every match from sources 2 and 3 (a single body may mention multiple versions — `0.0.6 and v0.0.10`). Then validate. -- **Clamp future:** if the parsed version is greater than `$LATEST`, treat it as unparseable. This catches roadmap labels that slipped past source 1. -- **Validate against tags:** confirm the parsed version exists as an actual git tag. This catches reporter typos such as `NemoClaw: v0.1.0` (no such release in the current `v0.0.x` line) and calver mistakes like `NemoClaw: 2026.3.11` (date string, not a tag). +**Validate against the tag list.** A parsed version must exist as a real git tag, otherwise drop it. This single check kills four classes of error in one pass: + +- Reporter typos that cite a non-existent version (`v0.1.0` when only `v0.0.x` is released — observed 3× in the live backlog). +- Calver mistakes (`2026.3.11` — observed 1×). +- Future roadmap labels that slipped past source 1. +- Versions parsed from prose that happen to look semver-ish but aren't releases. ```bash git ls-remote --tags --refs git@github.com:NVIDIA/NemoClaw.git \ - | awk -F/ '{print $NF}' \ - | grep -Fx "v$PARSED_VERSION" >/dev/null \ - || PARSED_VERSION="" # treat as unparseable + | awk -F/ '{print $NF}' > /tmp/nemoclaw-tags.txt + +# For each candidate version V: +grep -Fxq "$V" /tmp/nemoclaw-tags.txt || drop_version "$V" ``` -If no version survives both passes, drop the issue from the candidate set — we cannot establish "previous version". +After validation, **pick the smallest surviving version** as the reported version (most conservative — it maximizes versions-behind). This handles "this bug was first reported on v0.0.6 and still happens on v0.0.10" cleanly: we verify against latest, and if the bug is gone, both reports are addressed. -### Implementer note: regex-pipeline pitfall +If no version survives, drop the issue from the candidate set — we cannot establish "previous version". -In the v1 dry-run, a naive jq pipeline that chained the primary and fallback regexes via `[scan(primary)] | first | .[0] | tonumber // [scan(fallback)] | first | .[0] | tonumber` silently dropped 9 real candidates (e.g. #2861 with `NemoClaw 0.0.32` in body, #2604 with `NemoClaw: 0.0.28`). When the primary regex matched empty, `null | first` errored, and `//` did not propagate cleanly to the fallback. +### Implementer note: regex-pipeline pitfalls -Whichever language you implement in, structure the parser so the empty-match path returns null cleanly (not an error). Bind each pass to a named variable and `coalesce` them at the end: +Two real failure modes surfaced during the v1 dry-run. Test both before trusting your implementation: -```text -primary := first match of \bv\d+\.\d+\.\d+\b in body (or null) -fallback := first match of \b\d+\.\d+\.\d+\b on nemoclaw/version lines (or null) -result := primary ?? fallback -``` +1. **Empty-match handling.** A naive pipeline like `[scan(regex)] | first | .[0] | tonumber // fallback` silently dropped 9 real candidates (e.g. #2861 with `NemoClaw 0.0.32`, #2604 with `NemoClaw: 0.0.28`). When `scan` returns no matches, `[]` flows in, `first` returns null, `null | .[0]` errors, and `//` does not propagate cleanly through the error. Bind each pass to a named variable, coalesce at the end: + + ```text + primary := first nemoclaw-anchored match in body (or null) + result := primary ?? null + ``` + + Then explicitly test against a body with **no** version mention. + +2. **Capture-group consistency.** A regex without a capture group (e.g. `\bv\d+\.\d+\.\d+\b`) makes `scan` emit raw strings; with a capture group (e.g. `\b(v\d+\.\d+\.\d+)\b`), `scan` emits arrays. Mixing the two within one pipeline (`first | .[0]?`) works for one and silently fails for the other. Use capture groups consistently across all branches. -Always test against an issue body with **no** version mention before trusting the result — that's the path that exercises the empty-match handling. +3. **Variable scoping in `select(...)`.** A line like `select($tags | index(.))` rebinds `.` to `$tags` inside the parens, so `.` no longer refers to the surrounding label being checked. Bind first: `. as $lbl | select($tags | any(. == $lbl))`. Symptom in this dry-run: the future-release label `v0.0.35` passed validation that should have rejected it. --- From 10268c45f375ee0d21b8fd26bfcc99d1f59c6232 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Tue, 5 May 2026 16:25:04 -0700 Subject: [PATCH 05/40] fix(verify-stale): validate reproducer on baseline before trusting latest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A clean run on latest is ambiguous on its own: - Scenario A: bug really got fixed (script worked, latest passes) - Scenario B: script was junk all along (script never triggered the bug; latest passes for the same reason baseline would) These look identical from the latest pass alone, which means the previous flow could land `fixed-on-latest` on a script that never had a chance — a false positive that the +25 commit and +25 PR signals only partially mitigate. This change adds a baseline pass before the latest pass: - Step 8a: install reported version on the box. - Step 8b: run reproducer, compare output to issue's actual-result description. Match = script validated. - Step 8c: if no match (script silent OR errored on the wrong thing), synth-repro using the issue body PLUS the baseline transcript so the LLM can react to the actual failure mode. Apply -30. Retry. Still no match -> verify-inconclusive, skip latest entirely. - Step 8d: install latest, run validated reproducer. Score from this. A single trigger drives synthesis: "the script didn't expose the bug on baseline." Whether it was silent or noisy doesn't matter; both mean the script needs work. This collapses the previous separate "narrative-only -> synth" and "verbatim-failed -> ???" paths into one rule. Step 6 simplified accordingly: just extract verbatim if available, otherwise carry the issue body forward to Step 8b for on-demand synthesis. No more "give up before provisioning" branch — that decision moves into Step 8c where it has more context to work with. Step 9 adds a baseline-validation gating rule. If the reported- version install fails (old releases rot — installer URLs, deps, OS images drift), the score is capped at 84 unless commit-area or PR- mention evidence also fires. That forces the @-mention-reporter band when we lack independent corroboration, which is the honest position when we couldn't validate the script ourselves. Step 11 split into two failure types: - Latest-install fail or harness error -> hard infra failure (no label, optional comment, move on). - Baseline-install fail -> degraded mode (skip baseline gate, run latest anyway, apply Step 9 cap). Wallclock budget bumped from 15 to 25 minutes per verification to accommodate two installs. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 93 +++++++++++++++---- 1 file changed, 75 insertions(+), 18 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index aa3bee9a2a5..ce133dc54de 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -144,13 +144,12 @@ CPU default keeps cost low. Only escalate to GPU when the reproducer needs one. ## Step 6: Extract the Reproducer -Try in order, stop at the first that works: +Extract whatever's available from the issue body. The decision about *whether the reproducer is good enough* lives in Step 8 (validate-on-baseline), not here. -1. **Verbatim extraction:** the first fenced code block in the issue body that contains a `nemoclaw` invocation. No confidence penalty. -2. **LLM synthesis:** if no fenced block matches, synthesize a shell script from the narrative bug report. Apply a **−30 confidence penalty** later. -3. **Give up:** if neither produces a runnable script, mark the issue `verify-inconclusive`, post a short comment explaining why, and move on. Do not provision a Brev box. +1. **Verbatim:** the first fenced code block (triple-backtick or `<pre>`) containing a `nemoclaw` invocation. Save to `./reproducer.sh`. No confidence penalty (yet). +2. **No verbatim block found:** leave `./reproducer.sh` absent. Step 8b will synthesize from the issue body on demand and apply the **−30 synth penalty** at that point. -Save the chosen script to `./reproducer.sh`. Both verbatim and synthesized scripts will be quoted in the final comment as evidence. +The "give up immediately" path is gone. Synthesis happens at validation time so it has the baseline transcript to react to, not just the issue body in isolation. The give-up decision now lands in Step 8c when synth fails to produce a script that actually exposes the bug. --- @@ -210,28 +209,74 @@ fi trap '[ "$PROVISIONED_NEW" = "1" ] && brev delete "$INSTANCE_NAME" --yes || true' EXIT ``` -Wallclock cap per verification: **15 minutes** including reuse-check, install, and reproduction. If a provisioned box isn't ready in time, abort and treat as an infra failure (Step 11). +Wallclock cap per verification: **25 minutes** to accommodate two installs (reported version baseline + latest). If a provisioned box isn't ready in time, abort and treat as an infra failure (Step 11). --- -## Step 8: Reset, Install Latest, Run the Reproducer +## Step 8: Validate on Baseline, Verify on Latest -Even on a reused box, reset NemoClaw state before installing — hermeticity matters more than the few seconds saved. +Two-pass design. + +- **Baseline pass (8a–8c):** install the **reported version**, run the reproducer, confirm it actually exposes the bug as described. This is the gate that proves the script is real. +- **Latest pass (8d):** install **latest**, run the validated reproducer. This is what the confidence score is built on. + +Without the baseline gate, a clean run on latest is ambiguous: maybe the bug really got fixed, maybe the script was never capable of triggering it. The baseline disambiguates. + +### Step 8a: Install reported version + +```bash +RESET="rm -rf ~/.nemoclaw 2>/dev/null; sudo rm -f /usr/local/bin/nemoclaw 2>/dev/null; true" + +brev exec "$INSTANCE_NAME" "$RESET" + +# Install reported version. URL pattern may need adjustment per release line. +brev exec "$INSTANCE_NAME" "curl -fsSL https://nemoclaw.nvidia.com/install.sh | bash -s -- --version $REPORTED_VERSION" \ + || BASELINE_INSTALL_FAILED=1 +brev exec "$INSTANCE_NAME" "nemoclaw --version" +``` + +If install fails (old releases rot — installer URLs, deps, OS images all drift over time), set `BASELINE_INSTALL_FAILED=1` and **skip 8b/8c**, going straight to 8d. Note "baseline-install-skipped" in the final comment. Step 9's scoring rule handles the degraded mode. + +### Step 8b: Run reproducer on baseline, compare to issue symptom + +If `./reproducer.sh` exists (verbatim from Step 6), run it. Otherwise synth on demand from the issue body (apply −30 penalty now, locked in for the rest of the run). ```bash -# Reset prior NemoClaw state on the box (safe no-op on a fresh box). -brev exec "$INSTANCE_NAME" "rm -rf ~/.nemoclaw 2>/dev/null; sudo rm -f /usr/local/bin/nemoclaw 2>/dev/null; true" +brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh +brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcript.log +``` + +LLM compares `baseline-transcript.log` against the issue's "Actual result" / error description. + +- **Match** (script produced the bug as described): reproducer validated. Proceed to 8d. +- **No match** (silent pass, or wrong/unrelated error): script has gaps. Proceed to 8c. -# Install latest NemoClaw. +### Step 8c: Synth-repro and retry on baseline + +LLM rewrites `./reproducer.sh` using the full issue context (description, environment, symptoms) **plus the baseline transcript** so it can react to what actually happened. Apply **−30 confidence penalty** (or keep it if 8b already applied it for the missing-verbatim case). + +```bash +brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh +brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcript-2.log +``` + +- **Match:** validated (with −30 baked in). Proceed to 8d. +- **Still no match:** mark `verify-inconclusive`. Post a comment that includes both reproducer attempts and both baseline transcripts with the message "couldn't establish a working reproducer for this bug on `$REPORTED_VERSION`." **Skip 8d** — there's nothing to verify on latest. + +### Step 8d: Install latest, run validated reproducer + +```bash +brev exec "$INSTANCE_NAME" "$RESET" brev exec "$INSTANCE_NAME" "curl -fsSL https://nemoclaw.nvidia.com/install.sh | bash" brev exec "$INSTANCE_NAME" "nemoclaw --version" -# Copy and run the extracted reproducer; capture full transcript. brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh -brev exec "$INSTANCE_NAME" @reproducer-runner.sh 2>&1 | tee ./transcript.log +brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./latest-transcript.log ``` -If the install itself fails (e.g. installer regression — see #3058 for a current example), this is an **infra failure** — see Step 11. Do not score or label the issue. +If the install of **latest** fails (e.g. installer regression — see #3058 for a current example), this is an infra failure — see Step 11. Do not score or label the issue. + +If install succeeds, `latest-transcript.log` is the input to Step 9 scoring. For interactive debugging when something looks off: @@ -247,14 +292,16 @@ Start at 0. Apply each rule that fires. | Signal | Delta | |---|---| -| Reproducer ran cleanly on latest, exit 0, expected output observed | +50 | +| Reproducer ran cleanly on **latest** (8d), exit 0, no bug symptom observed | +50 | | Commits between reported version and `$LATEST` touch the implicated component (`git log v<reported>..$LATEST -- <path>`) | +25 | | A merged PR mentions this issue number or its symptom | +25 | -| Reproducer was LLM-synthesized, not extracted verbatim | −30 | -| Any partial error, warning, or flaky behavior in the repro run | −50 | +| Reproducer was LLM-synthesized at any point (Step 8b synth or Step 8c retry) | −30 | +| Any partial error, warning, or flaky behavior in the latest run (8d) | −50 | Total is clamped to `[0, 100]`. +**Baseline-validation gating.** The +50 weight assumes the reproducer was *validated* — i.e., it produced the bug symptom on baseline (Step 8b/8c match). If `BASELINE_INSTALL_FAILED=1` (Step 8a fall-through, baseline pass skipped), the +50 still applies but **cap the total at 84** unless commits-touched-area or merged-PR-mention also fires. Without baseline AND without corroborating evidence, the cleanest landing is the 60–84 band where the reporter is asked to confirm — we don't have enough on our own to claim ≥85. + **Action:** | Score | Label | Comment | @@ -323,7 +370,9 @@ gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "fixed-on-lates ## Step 11: Infra Failure Handling -If reuse-check, provisioning, install, or the test harness itself fails (not the reproducer): +Two different failure types, two different responses. + +**Latest-install failure** (Step 8d) or reuse-check / provisioning / harness errors: hard infra failure. - Print the error. - Apply **no label** — infra failures must not pollute the verification record. @@ -332,6 +381,14 @@ If reuse-check, provisioning, install, or the test harness itself fails (not the The next weekly run retries naturally. +**Baseline-install failure** (Step 8a, reported version won't install on a modern image): not a hard failure — degraded mode. + +- Set `BASELINE_INSTALL_FAILED=1`, skip 8b/8c, jump to 8d. +- Step 9 applies the score cap (max 84) unless corroborating evidence fires. +- Note "baseline-install-skipped" in the final comment so a reviewer knows the verification ran without the script-validation gate. + +This degradation is expected — old releases rot. We still want to extract whatever signal we can from the latest run plus PR/commit evidence, just at a more conservative confidence ceiling. + --- ## Step 12: Log to Activity From ca865ada855902e0b7ee583588cf66e91b499526 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Tue, 5 May 2026 16:41:42 -0700 Subject: [PATCH 06/40] fix(verify-stale): close gaps surfaced before E2E run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tier-1 (blocking for E2E): - Add the still-reproduces-on-latest path. If the latest run output matches the issue symptom, the bug is still live — the skill posts a "still reproducible" comment with both transcripts, applies no label, and includes a dated marker `<!-- ... v1 YYYY-MM-DD -->`. No new label per user direction; a 7-day TTL on the marker handles re-verification on the next weekly cron. - Update the comment template for the two-pass flow. The previous template described a single transcript; the new one has Baseline and Latest sections plus a dedicated still-reproduces template, and the idempotency marker now carries today's date. - Update the Step 12 log template to record baseline-install, baseline-match, latest-install, and latest-result independently. - Clarify in Step 4 that REPORTED_VERSION must be the full tag string ("v0.0.32"), not just the patch number (32). Step 8a's installer expects the full tag. Tier-2: - Spell out an explicit match rubric in Step 8b: exit code agreement, symptom-phrase match (LLM-judged semantic equivalence counts), and distinguishing the bug from infra noise (DNS / auth / rate limits). - Replace the minimal `rm -rf ~/.nemoclaw` reset with a comprehensive one that stops nemoclaw/openshell processes, removes spawned Docker containers, frees common ports (8080, 18789, 9119), and removes the installed binaries and lib paths. Run before each install in 8a and 8d; idempotent so it's safe on a fresh box. - Add interactive-subcommand handling in 8b: auto-detect `nemoclaw onboard` / `configure` and try `--non-interactive`, then `--dangerously-skip-prompts`, then stdin pre-feed. Fall through to Step 8c (synth) if none work. Tier-3 / opportunistic: - Fix the actual installer command (gap 9). The installer accepts the target ref via `NEMOCLAW_INSTALL_TAG` env var, NOT a `--version` flag (verified against install.sh source). Updated 8a accordingly. - Update Step 3 idempotency: drop on either label OR a marker comment within the last 7 days. Previously a marker excluded the issue forever; the release sweep only clears labels, so the marker alone made re-verification impossible. The 7-day TTL also handles the still-reproduces case naturally. - Extend wallclock budget to 60 min when issue body contains time-sensitive keywords (`after N minutes`, `eventually`, `memory leak`, etc.). Hard ceiling at 60 — bugs that require hours fall out of v1 scope. - Keep provisioned boxes for 30 minutes on verify-inconclusive outcomes so a maintainer can `brev shell` in and triage. Reused boxes always stay. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 151 +++++++++++++++--- 1 file changed, 128 insertions(+), 23 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index ce133dc54de..f7a8db2dc72 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -70,7 +70,10 @@ Apply these rules in order. Drop any issue that fails a rule. **Component allowlist (must have at least one):** `NemoClaw CLI`, `Sandbox`, `OpenShell`, `Docker`, `Getting Started`, or any `Platform:` label that survived the platform skip. -**Idempotency:** drop if any comment body on the issue contains `<!-- nemoclaw-verify-stale v1 -->`. The skill never re-verifies an issue within the same release window. (The release sweep in `nemoclaw-maintainer-cut-release-tag` clears prior `fixed-on-latest` and `verify-inconclusive` labels on each release, which is what re-opens the candidate set.) +**Idempotency:** drop if **either** of these is true: + +- The issue carries a `fixed-on-latest` or `verify-inconclusive` label. (Cleared by the release sweep in `nemoclaw-maintainer-cut-release-tag` so the issue re-opens on each release.) +- A `<!-- nemoclaw-verify-stale v1 YYYY-MM-DD -->` comment was posted **within the last 7 days**. The marker carries a date so the candidate filter can apply a TTL — useful for the still-reproduces case (Step 9), where no label is applied and we want next week's run to re-verify rather than skip forever. **Candidate rule:** keep the issue if **either**: @@ -112,6 +115,8 @@ After validation, **pick the smallest surviving version** as the reported versio If no version survives, drop the issue from the candidate set — we cannot establish "previous version". +**Variable format for downstream steps.** Set `REPORTED_VERSION` to the **full tag string** (e.g., `REPORTED_VERSION="v0.0.32"`), not just the patch number. Step 8a's installer expects the full tag via the `NEMOCLAW_INSTALL_TAG` env var. + ### Implementer note: regex-pipeline pitfalls Two real failure modes surfaced during the v1 dry-run. Test both before trusting your implementation: @@ -211,6 +216,8 @@ trap '[ "$PROVISIONED_NEW" = "1" ] && brev delete "$INSTANCE_NAME" --yes || true Wallclock cap per verification: **25 minutes** to accommodate two installs (reported version baseline + latest). If a provisioned box isn't ready in time, abort and treat as an infra failure (Step 11). +**Extended budget for time-sensitive bugs.** If the issue body contains keywords suggesting the bug only manifests over time (`after N minutes`, `after N requests`, `eventually`, `over time`, `memory leak`, `long-running`, `idle for`), bump the cap to **60 minutes**. Detection is simple keyword match. Hard ceiling at 60 min — bugs that genuinely require hours fall out of v1 scope. + --- ## Step 8: Validate on Baseline, Verify on Latest @@ -222,15 +229,36 @@ Two-pass design. Without the baseline gate, a clean run on latest is ambiguous: maybe the bug really got fixed, maybe the script was never capable of triggering it. The baseline disambiguates. -### Step 8a: Install reported version +### Comprehensive reset (run before each install) + +NemoClaw spawns OpenShell sandboxes (containers), runtime services, and listening processes. A naive `rm -rf ~/.nemoclaw` doesn't clean those — the latest install would inherit baseline state and contaminate the result. Use this fuller reset between installs: ```bash -RESET="rm -rf ~/.nemoclaw 2>/dev/null; sudo rm -f /usr/local/bin/nemoclaw 2>/dev/null; true" +RESET=$(cat <<'SCRIPT' +nemoclaw destroy --all --force 2>/dev/null || true +pkill -9 -f nemoclaw 2>/dev/null || true +pkill -9 -f openshell 2>/dev/null || true +docker ps -a --filter "name=openshell-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true +docker ps -a --filter "name=nemoclaw-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true +rm -rf ~/.nemoclaw 2>/dev/null +sudo rm -f /usr/local/bin/nemoclaw 2>/dev/null +sudo rm -rf /usr/local/lib/nemoclaw 2>/dev/null +for port in 8080 18789 9119; do fuser -k -n tcp $port 2>/dev/null || true; done +true +SCRIPT +) +``` + +Idempotent — fails silently when there's nothing to clean. Run via `brev exec "$INSTANCE_NAME" "$RESET"` before 8a's install and again before 8d's install. +### Step 8a: Install reported version + +The installer accepts the target ref via the `NEMOCLAW_INSTALL_TAG` env var (verified against `install.sh` source — defaults to `latest` if unset). It is **not** a `--version` flag. + +```bash brev exec "$INSTANCE_NAME" "$RESET" -# Install reported version. URL pattern may need adjustment per release line. -brev exec "$INSTANCE_NAME" "curl -fsSL https://nemoclaw.nvidia.com/install.sh | bash -s -- --version $REPORTED_VERSION" \ +brev exec "$INSTANCE_NAME" "NEMOCLAW_INSTALL_TAG=$REPORTED_VERSION bash -c 'curl -fsSL https://nemoclaw.nvidia.com/install.sh | bash'" \ || BASELINE_INSTALL_FAILED=1 brev exec "$INSTANCE_NAME" "nemoclaw --version" ``` @@ -241,15 +269,27 @@ If install fails (old releases rot — installer URLs, deps, OS images all drift If `./reproducer.sh` exists (verbatim from Step 6), run it. Otherwise synth on demand from the issue body (apply −30 penalty now, locked in for the rest of the run). +**Interactive subcommand handling.** Many `nemoclaw onboard` / `nemoclaw configure` invocations prompt for input and will hang in a non-interactive shell. Auto-detect such subcommands in the script and apply, in order: + +1. Add `--non-interactive` if the version supports it. +2. Add `--dangerously-skip-prompts` (issue #2168 confirmed this exists for at least some Jetson paths). +3. Pre-feed answers via stdin: `printf 'yes\n\n\n' | nemoclaw onboard ...` + +If none work, route the script to Step 8c (synth-repro) so the LLM can rewrite it using non-interactive equivalents. + ```bash brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcript.log ``` -LLM compares `baseline-transcript.log` against the issue's "Actual result" / error description. +**Match rubric.** LLM compares `baseline-transcript.log` to the issue's "Actual result" / error description. Match criteria: -- **Match** (script produced the bug as described): reproducer validated. Proceed to 8d. -- **No match** (silent pass, or wrong/unrelated error): script has gaps. Proceed to 8c. +1. **Exit code agrees** with what the issue describes (non-zero if issue describes a failure, zero if issue describes a wrong-output bug). Necessary but not sufficient. +2. **Symptom phrase match:** transcript contains a key error phrase from the issue (e.g., issue says `Permission denied on generate-openclaw-config.py`, transcript says `EACCES: permission denied, open '...generate-openclaw-config.py'` — semantic equivalence counts). +3. **Distinguish bug from infra noise:** generic network / DNS / auth errors don't count as a match unless the issue itself describes them. A bug about config parsing that fails at "could not resolve nvidia.com" is an infra failure, not a reproduction. + +- **Match** → reproducer validated. Proceed to 8d. +- **No match** (silent pass, wrong error, or infra noise): script has gaps. Proceed to 8c. ### Step 8c: Synth-repro and retry on baseline @@ -302,7 +342,7 @@ Total is clamped to `[0, 100]`. **Baseline-validation gating.** The +50 weight assumes the reproducer was *validated* — i.e., it produced the bug symptom on baseline (Step 8b/8c match). If `BASELINE_INSTALL_FAILED=1` (Step 8a fall-through, baseline pass skipped), the +50 still applies but **cap the total at 84** unless commits-touched-area or merged-PR-mention also fires. Without baseline AND without corroborating evidence, the cleanest landing is the 60–84 band where the reporter is asked to confirm — we don't have enough on our own to claim ≥85. -**Action:** +**Action (when latest run was clean — bug not reproduced):** | Score | Label | Comment | |---|---|---| @@ -310,7 +350,16 @@ Total is clamped to `[0, 100]`. | 60–84 | `fixed-on-latest` | Evidence-rich, **@-mention the original reporter** to confirm. | | <60 | `verify-inconclusive` | Short, honest "couldn't verify" explanation. | -The skill **never closes issues**. A maintainer pulls that trigger after reviewing the label and comment. +**Special case: latest output matches the issue symptom (bug still reproduces on latest).** + +This is not a flake — the skill positively confirmed the bug is still live. Don't apply the +50 weight (the bug isn't fixed) and skip the score table entirely. + +- Post a "still reproduces on latest" comment with both transcripts. +- Apply **no label**. +- Include the marker `<!-- nemoclaw-verify-stale v1 YYYY-MM-DD -->` with today's date so the candidate filter applies the 7-day TTL (Step 3 idempotency). +- Next weekly run picks the issue back up after the TTL — if the bug gets fixed in the meantime, that run catches it. + +The skill **never closes issues** in any branch. A maintainer pulls that trigger after reviewing the label and comment. --- @@ -322,27 +371,46 @@ The skill **never closes issues**. A maintainer pulls that trigger after reviewi - URLs containing `@` (basic-auth credentials). - File paths under the reporter's home directory (replace with `~/`). -**Comment template:** +**Comment template (fixed / inconclusive — bug not reproduced on latest):** ````markdown ## Stale-issue verification — automated **Reported on:** v0.0.31 -**Verified on:** v0.0.35 (commit abc1234) +**Verified on:** v0.0.34 (commit abc1234) **Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 / <CUDA version if GPU> -**Reproducer source:** extracted verbatim from issue body | LLM-synthesized from narrative -**Result:** not reproducible — exit 0, expected output observed. -**Confidence:** 88 / 100. Labelling `fixed-on-latest`. +### Baseline (reported version) + +- Install: succeeded · skipped (install rotted) +- Reproducer: extracted verbatim · synthesized (−30 penalty) +- Result: bug symptom matched (validated) · could not validate (skipped Step 8c gate) -<details><summary>Reproduction transcript</summary> +<details><summary>Baseline transcript</summary> ```text -<full transcript here> +<full baseline transcript> ``` </details> +### Latest + +- Install: succeeded +- Result: not reproducible — clean run, no bug symptom observed + +<details><summary>Latest transcript</summary> + +```text +<full latest transcript> +``` + +</details> + +### Verdict + +**Confidence:** 88 / 100. Labelling `fixed-on-latest`. + <details><summary>Relevant changes since v0.0.31</summary> - abc1234 — fix: <commit subject> @@ -352,10 +420,42 @@ The skill **never closes issues**. A maintainer pulls that trigger after reviewi If this verification is wrong, please reopen the issue with a comment and the skill will re-verify on the next release. -<!-- nemoclaw-verify-stale v1 --> +<!-- nemoclaw-verify-stale v1 2026-05-12 --> +```` + +**Comment template (still reproduces — Step 9 special case):** + +````markdown +## Stale-issue verification — still reproducible + +**Reported on:** v0.0.31 +**Verified on:** v0.0.34 (commit abc1234) +**Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 + +The skill ran the reported reproducer on v0.0.34 and observed the same bug symptom described in this issue. The bug is still live. + +No label applied. Will re-verify automatically next weekly run; if a fix lands in the interim, the next pass catches it. + +<details><summary>Baseline transcript (validated reproducer)</summary> + +```text +<baseline transcript> +``` + +</details> + +<details><summary>Latest transcript (bug still observed)</summary> + +```text +<latest transcript> +``` + +</details> + +<!-- nemoclaw-verify-stale v1 2026-05-12 --> ```` -The trailing HTML comment is the **idempotency marker** Step 3 looks for. Never omit it. +The trailing HTML comment is the **idempotency marker** Step 3 looks for. Always include today's date in `YYYY-MM-DD` format so the candidate filter can apply the 7-day TTL. **Post the comment and apply the label:** @@ -389,6 +489,8 @@ The next weekly run retries naturally. This degradation is expected — old releases rot. We still want to extract whatever signal we can from the latest run plus PR/commit evidence, just at a more conservative confidence ceiling. +**Keep-box-on-inconclusive.** When `verify-inconclusive` lands (Step 8c gave up, or Step 9 score < 60), **delay the cleanup `brev delete` by 30 minutes** if the box was provisioned by this run. Print the `brev shell "$INSTANCE_NAME"` command in the run output so a maintainer can hop in and triage. Reused boxes stay regardless. Ship-failed verifications are the exact case where having an inspectable artifact pays for itself. + --- ## Step 12: Log to Activity @@ -399,12 +501,15 @@ After each issue (verified, inconclusive, or infra-failed), append to `~/develop ### NVIDIA/NemoClaw#<number> — <title> **Date:** YYYY-MM-DD **Reported on:** v0.0.31 -**Verified on:** v0.0.35 +**Verified on:** v0.0.34 **Environment:** CPU | GPU (<instance type>) **Box:** reused <name> | provisioned <name> -**Reproducer:** verbatim | synthesized | none -**Confidence:** 88 / 100 -**Label applied:** fixed-on-latest | verify-inconclusive | none (infra) +**Baseline install:** succeeded | failed (degraded mode) +**Baseline match:** validated (verbatim) | validated (synth) | failed (verify-inconclusive) | skipped +**Latest install:** succeeded | failed (infra error) +**Latest result:** not-reproduced (clean) | still-reproduces | partial / flake | n/a (skipped 8d) +**Confidence:** 88 / 100 | n/a (still-reproduces) +**Label applied:** fixed-on-latest | verify-inconclusive | none (still-reproduces) | none (infra) **Brev wall time (approx):** N min --- From 2c7e90678247a89f44269368cf2f58769084107a Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Wed, 6 May 2026 14:48:08 -0700 Subject: [PATCH 07/40] fix(verify-stale): close gaps 1, 4, 5, 6, 7 before E2E MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five concrete fixes from the pre-E2E gap analysis, plus a verified finding on gap 2 that lowered its risk profile. Gap 1 — sudo behavior. The reset script's `sudo rm -f` and `sudo rm -rf` could hang on a password prompt if the Brev image's default user lacks passwordless sudo. Switched all sudo calls to `sudo -n` (non- interactive) so they fail fast instead. Added a precondition note explaining the assumption: Brev stock images allow passwordless sudo, custom images may not. The user-local install path (~/.nemoclaw) is fully reset regardless of sudo state, so the worst case is a stale /usr/local/bin/nemoclaw symlink that the next install overwrites. Gap 2 — verified the install architecture supports old-version pinning natively. The bootstrap clones the requested ref and runs that tag's own install.sh (with a payload-marker fallback for legacy ones), so NEMOCLAW_INSTALL_TAG=v0.0.32 works architecturally. Risk reduces to "does v0.0.32's own installer still work on a 2026 OS image" — handled by Step 11's degraded-mode fall-through. No SKILL.md change needed. Gap 4 — path extraction was hand-waved. The +25 commits-touched-area weight referenced `git log v<reported>..$LATEST -- <path>` without specifying how to determine `<path>`. Added a three-tier extraction procedure: stack-trace path mentions parsed from the body and mapped to repo paths, then a component-label-to-directory map (NemoClaw CLI → bin/, Sandbox → src/lib/sandbox/, etc.), then title-keyword fallbacks. If none yields a path, skip the +25 signal entirely rather than guessing — floating the weight would inflate scores meaninglessly. Gap 5 — PR-search query was hand-waved. The +25 PR-mention weight had no concrete query. Added two-stage search: first `gh pr list --search "$ISSUE_NUMBER"` filtered for actual `#NNNN` references in body or title, then a symptom-phrase fallback if direct reference returns nothing. PRs that merged before the issue was filed are excluded via `mergedAt > tag-date(REPORTED_VERSION)`. Gap 6 — issues without an "Actual result" section had no match path. Behavioral / configuration bugs (e.g., #1242 "should default to a stable released version") describe a wrong default rather than a runtime error. Added a fallback to Step 8b's match rubric: use the title + full body as the symptom, match if reproducer's outcome contradicts the issue's expected behavior. If neither error string nor expected-behavior contradiction can be identified, route to Step 8c (synth-repro) so the LLM produces a more diagnostic script. Gap 7 — transcript redaction was thin. Step 10 only covered tokens, paths, and basic-auth URLs. Real transcripts leak more: Authorization headers, JWTs, GitHub PATs, AWS keys, NVIDIA API keys, internal hostnames, emails (PII), long base64 blobs. Expanded the redaction table with concrete patterns for each. Added an order note: longest/most-specific patterns first so the generic base64 catchall doesn't mask what was actually redacted. Made it explicit that redaction runs on every quoted chunk in the comment, not just issue- body excerpts. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 84 ++++++++++++++++--- 1 file changed, 74 insertions(+), 10 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index f7a8db2dc72..8ce61418e95 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -241,8 +241,8 @@ pkill -9 -f openshell 2>/dev/null || true docker ps -a --filter "name=openshell-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true docker ps -a --filter "name=nemoclaw-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true rm -rf ~/.nemoclaw 2>/dev/null -sudo rm -f /usr/local/bin/nemoclaw 2>/dev/null -sudo rm -rf /usr/local/lib/nemoclaw 2>/dev/null +sudo -n rm -f /usr/local/bin/nemoclaw 2>/dev/null || true +sudo -n rm -rf /usr/local/lib/nemoclaw 2>/dev/null || true for port in 8080 18789 9119; do fuser -k -n tcp $port 2>/dev/null || true; done true SCRIPT @@ -251,6 +251,8 @@ SCRIPT Idempotent — fails silently when there's nothing to clean. Run via `brev exec "$INSTANCE_NAME" "$RESET"` before 8a's install and again before 8d's install. +**Sudo precondition.** All `sudo` invocations use `sudo -n` (non-interactive) so they fail fast instead of hanging on a password prompt. The skill assumes the Brev image's default user has passwordless sudo configured — Brev's stock images do; custom images may not. If `sudo -n` fails, the binary cleanup is best-effort and a stale `/usr/local/bin/nemoclaw` may persist. The user-local install path (`~/.nemoclaw`) is fully reset regardless. + ### Step 8a: Install reported version The installer accepts the target ref via the `NEMOCLAW_INSTALL_TAG` env var (verified against `install.sh` source — defaults to `latest` if unset). It is **not** a `--version` flag. @@ -282,14 +284,20 @@ brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcript.log ``` -**Match rubric.** LLM compares `baseline-transcript.log` to the issue's "Actual result" / error description. Match criteria: +**Match rubric.** LLM compares `baseline-transcript.log` to the issue's "Actual result" / error description. Match criteria, in order: 1. **Exit code agrees** with what the issue describes (non-zero if issue describes a failure, zero if issue describes a wrong-output bug). Necessary but not sufficient. 2. **Symptom phrase match:** transcript contains a key error phrase from the issue (e.g., issue says `Permission denied on generate-openclaw-config.py`, transcript says `EACCES: permission denied, open '...generate-openclaw-config.py'` — semantic equivalence counts). 3. **Distinguish bug from infra noise:** generic network / DNS / auth errors don't count as a match unless the issue itself describes them. A bug about config parsing that fails at "could not resolve nvidia.com" is an infra failure, not a reproduction. +**Fallback for issues without an explicit "Actual result" section.** Many bug reports describe a *behavioral* problem rather than a runtime error — e.g., "should default to a stable released version" (#1242), "configuration is not persisted across rebuilds" (#3030). These have no comparable error string. In that case: + +1. Use the issue's **full title + description** as the symptom signal. +2. Match if the reproducer's outcome **contradicts the issue's stated expected behavior** (or matches the stated wrong behavior). E.g., issue says "expected: stable release; actual: nightly", reproducer prints `nightly-build-2026.04.x` → that's a match. +3. If neither error string nor expected-behavior contradiction can be identified, route the script to Step 8c (synth-repro) — let the LLM produce a more diagnostic script that emits something testable. + - **Match** → reproducer validated. Proceed to 8d. -- **No match** (silent pass, wrong error, or infra noise): script has gaps. Proceed to 8c. +- **No match** (silent pass, wrong error, infra noise, or no testable outcome): script has gaps. Proceed to 8c. ### Step 8c: Synth-repro and retry on baseline @@ -333,13 +341,55 @@ Start at 0. Apply each rule that fires. | Signal | Delta | |---|---| | Reproducer ran cleanly on **latest** (8d), exit 0, no bug symptom observed | +50 | -| Commits between reported version and `$LATEST` touch the implicated component (`git log v<reported>..$LATEST -- <path>`) | +25 | -| A merged PR mentions this issue number or its symptom | +25 | +| Commits between reported version and `$LATEST` touch the implicated component (see "Path extraction" below) | +25 | +| A merged PR mentions this issue number or its symptom (see "PR search" below) | +25 | | Reproducer was LLM-synthesized at any point (Step 8b synth or Step 8c retry) | −30 | | Any partial error, warning, or flaky behavior in the latest run (8d) | −50 | Total is clamped to `[0, 100]`. +### Path extraction (for the +25 commits signal) + +The skill needs to know *which* path to `git log v<reported>..$LATEST -- <path>` against. Apply in order, stop at the first that yields a non-empty path: + +1. **Stack trace / file path mentions in the issue body.** Grep the body for absolute paths under known install roots, then map to repo paths: + - `/usr/local/lib/nemoclaw/<rel>` → `<rel>` in repo (e.g., `scripts/generate-openclaw-config.py`) + - `/usr/local/bin/nemoclaw*` → `bin/` + - `~/.nemoclaw/<rel>` → most often runtime state, drop unless the bug is config-related → `src/lib/config/` + - In-repo paths (e.g., `bin/lib/policies.js` mentioned literally) → use as-is +2. **Component-label-to-directory map.** Pick the first match: + - `NemoClaw CLI` → `bin/`, `src/` + - `Sandbox` → `src/lib/sandbox/`, `nemoclaw/sandbox/` + - `OpenShell` → `nemoclaw/openshell/`, `src/lib/openshell/` + - `Docker` → `Dockerfile`, `scripts/install-openshell.sh` + - `Getting Started` → `docs/`, `install.sh` + - `Integration: <X>` (when not in skip list) → `src/lib/integrations/<x>/` +3. **Title keywords.** "TUI" → `src/tui/`, "policy" → `src/lib/policy/`, "inference" → `src/lib/inference/`. + +If none of the above produces a path, **skip the +25 signal entirely** rather than guessing. Floating the +25 on every issue would inflate scores meaninglessly. + +### PR search (for the +25 PR signal) + +```bash +# Direct issue-number reference (covers most cases — "fixes #2861" etc.) +DIRECT_REF=$(gh pr list --repo NVIDIA/NemoClaw --state merged \ + --search "$ISSUE_NUMBER" \ + --json number,title,mergedAt,body \ + -q "[.[] | select((.body + \" \" + .title) | test(\"#$ISSUE_NUMBER\\\\b\"))]") + +# Symptom-phrase fallback (only if direct reference returns nothing) +if [ -z "$DIRECT_REF" ] || [ "$DIRECT_REF" = "[]" ]; then + SYMPTOM=$(extract first key error/symptom phrase from issue body, ~3-6 words) + SYMPTOM_REF=$(gh pr list --repo NVIDIA/NemoClaw --state merged \ + --search "\"$SYMPTOM\"" \ + --json number,title,mergedAt) +fi +``` + +Apply +25 if either query returns at least one PR with `mergedAt` strictly after the tag date of `$REPORTED_VERSION` (look up via `git log -1 --format=%cI v$REPORTED_VERSION`). PRs merged before the reporter even filed the issue can't have fixed it. + +If neither query returns anything, **skip the +25 signal**. + **Baseline-validation gating.** The +50 weight assumes the reproducer was *validated* — i.e., it produced the bug symptom on baseline (Step 8b/8c match). If `BASELINE_INSTALL_FAILED=1` (Step 8a fall-through, baseline pass skipped), the +50 still applies but **cap the total at 84** unless commits-touched-area or merged-PR-mention also fires. Without baseline AND without corroborating evidence, the cleanest landing is the 60–84 band where the reporter is asked to confirm — we don't have enough on our own to claim ≥85. **Action (when latest run was clean — bug not reproduced):** @@ -365,11 +415,25 @@ The skill **never closes issues** in any branch. A maintainer pulls that trigger ## Step 10: Compose and Post the Comment -**Redaction pass before posting.** Strip from any text quoted out of the issue body: +**Redaction pass before posting.** Run on **every** chunk of text quoted in the comment — issue body excerpts, baseline transcript, latest transcript, synth-repro scripts. Replace each match with `[REDACTED]`. The transcripts especially leak — they include full stdout/stderr from real installs and runs. -- Anything matching `(?i)(token|secret|password|api[_-]?key|bearer)[^\n]*[:=][^\n]*` -- URLs containing `@` (basic-auth credentials). -- File paths under the reporter's home directory (replace with `~/`). +| Pattern | Targets | +|---|---| +| `(?i)(token\|secret\|password\|api[_-]?key\|bearer)[^\n]*[:=][^\n]*` | Inline credentials in env/config/log output | +| `(?i)authorization:\s*\S+` | HTTP auth headers (often Bearer + JWT) | +| `eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}` | JWT tokens | +| `gh[pousr]_[A-Za-z0-9]{36,}` | GitHub PATs / install tokens | +| `AKIA[0-9A-Z]{16}` | AWS access key IDs | +| `(?i)aws_secret_access_key\s*=\s*\S+` | AWS secret keys | +| `(?i)nvapi-[A-Za-z0-9_-]{20,}` | NVIDIA API keys (NIM / build.nvidia.com) | +| URLs containing `@` before the host (e.g., `https://user:pw@host/...`) | Basic-auth credentials in URLs | +| `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}` | Email addresses (PII) | +| `\b[A-Za-z0-9+/]{60,}={0,2}\b` | Long base64 blobs (likely keys/sessions; tune length to taste — too short hits legit data) | +| `\b\w+\.(nvidia\.internal\|nv-internal\.com\|nvidia\.dev)\b` | Internal hostnames (extend list per team) | + +**File paths under the reporter's home directory** (`/Users/<name>/`, `/home/<name>/`) → replace with `~/`. Catches incidental username PII. + +**Order matters.** Run the longest, most-specific patterns first (JWT, AWS, NVIDIA-API) before the generic base64 catchall, otherwise the catchall masks the specific match and you lose the fact that *what* was redacted was a JWT vs a session blob. **Comment template (fixed / inconclusive — bug not reproduced on latest):** From ca50db5940e1fb144a03be24f214df22227d698c Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Wed, 6 May 2026 15:46:14 -0700 Subject: [PATCH 08/40] feat(verify-stale): add preconditions, local-first short-circuit, real SKU pick MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Preflight Brev auth and install-URL reachability before paying any cost (Step 6.5). For pure-CLI bugs with no sandbox, Docker, or GPU dependency, try the reproducer locally on the maintainer's `nemoclaw` install before provisioning a Brev box (Step 6.7) — same evidence, zero cost. Replace the `<your-team's-CPU-SKU>` placeholder in Step 7 with a runtime pick via `brev search cpu --sort price --json | jq` so the skill keeps working as SKUs change, with a `VERIFY_STALE_CPU_TYPE` env override for teams that need to pin. Drop the spurious `--yes` flag from `brev delete` — it does not exist on that subcommand and would error. `brev delete` is non-interactive by default. Print the instance name before the trap registers so manual cleanup is possible if the trap doesn't fire. Thread `INSTALL_URL` through Step 8a/8d so the URL override from Step 6.5 applies to both the baseline and latest installs. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 78 +++++++++++++++++-- 1 file changed, 70 insertions(+), 8 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 8ce61418e95..3a30037291d 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -158,13 +158,70 @@ The "give up immediately" path is gone. Synthesis happens at validation time so --- +## Step 6.5: Verify Preconditions + +Confirm `brev` is authenticated and the install URL resolves before paying any cost. Credentials live in `~/.brev/credentials.json` and are reused across shells under the same OS user, so once authenticated the auth check is a no-op until the token expires. + +```bash +# Brev auth — short-circuit only after the auth check, not before. +brev ls --json >/dev/null 2>&1 || { + echo "Brev not authenticated. Choose one:" + echo " 1) brev login --skip-browser # prints a URL, works from any shell" + echo " 2) brev login # opens browser, run in a separate terminal if your shell lacks a TTY" + echo " 3) brev login --token \"\$BREV_API_TOKEN\" # non-interactive, same env var used by test/e2e/brev-e2e.test.ts" + exit 1 +} + +# Install URL reachable — fails fast instead of mid-Brev-run if the host is down or the URL changed. +INSTALL_URL=${NEMOCLAW_INSTALL_URL:-https://nemoclaw.nvidia.com/install.sh} +curl -fsI "$INSTALL_URL" >/dev/null 2>&1 || { + echo "ERROR: install URL not reachable: $INSTALL_URL" + echo "Set NEMOCLAW_INSTALL_URL or check https://nemoclaw.nvidia.com is up." + exit 1 +} +``` + +If invoked from an environment without a TTY (some agent harnesses), prefer `brev login --skip-browser` or `--token` over the default browser flow. + +--- + +## Step 6.7: Try Local Reproduction First + +For pure-CLI reproducers (no sandbox state, no GPU, no integration tokens), try locally before paying for a Brev box. The evidence is identical — `nemoclaw <args>` on a maintainer laptop produces the same exit code and stdout as on a fresh Brev VM, modulo platform differences — and the run is free. + +**Predicate** — local-first applies if **all** of these hold: + +- Reproducer is a sequence of `nemoclaw <args>` invocations only. No `docker`, `kubectl`, `curl`, `npm`, networking setup, or filesystem fixtures. +- Issue has no `Sandbox`-only or `Docker` label and no GPU signal from Step 5. +- `which nemoclaw` resolves on the maintainer's machine and `nemoclaw --version` reports a build at or past `$LATEST` (a build between `$LATEST` and `$LATEST+main` is fine — these only differ by unmerged WIP). +- Maintainer is on Linux or macOS. Windows local repros are out of scope (per Step 3 platform skip rules). + +**If the predicate fires:** + +```bash +LOCAL_VERSION=$(nemoclaw --version 2>&1) +LOCAL_TRANSCRIPT=$(mktemp) +{ time bash reproducer.sh; } >"$LOCAL_TRANSCRIPT" 2>&1 +LOCAL_EXIT=$? +echo "Local: $LOCAL_VERSION, exit $LOCAL_EXIT" +``` + +Compare local result to the issue's "Actual Result" section using the same match rubric Step 8b applies on baseline: + +- **Local matches the issue symptom exactly** (same exit code + same diagnostic output) AND the symptom is the post-fix expected output → skip Brev. Use the local transcript as the verified-on-latest evidence. Step 10's comment must say `Environment: local install (<version>) — Brev provisioning skipped, outcome deterministic from CLI surface alone`. +- **Local result differs from the reported "Actual Result"** → continue to Step 7 and run on Brev. The local environment may be a confound (different OS, dirty config, partial build); remote confirms. +- **Local repro errors out for environmental reasons** (`nemoclaw: command not found`, npm link broken) → continue to Step 7. Treat as inconclusive locally, not a verification failure. + +**If the predicate does not fire:** proceed to Step 7 normally. Most sandbox-touching bugs need Brev. + +--- + ## Step 7: Reuse or Provision a Brev Box The skill prefers reuse over provisioning. A pool of `verify-stale-*` boxes (CPU and GPU) can be kept warm; reuse the matching one if available, otherwise provision. ```bash -# Ensure an active Brev session. brev ls fails if not authenticated. -brev ls --json >/dev/null 2>&1 || brev login +# Auth + install URL already verified by Step 6.5 — no need to re-check or auto-login here. # Determine class from Step 5: "cpu" or "gpu" INSTANCE_CLASS="cpu" # or "gpu" @@ -201,9 +258,12 @@ else # (>=20GB VRAM, >=500GB disk, compute >=8.0). Override with --type if needed. brev create "$INSTANCE_NAME" else - # CPU case: pass an explicit --type from your team's allowed CPU SKUs - # (brev create defaults to GPU). Pin this in your team config. - brev create "$INSTANCE_NAME" --type "<your-team's-CPU-SKU>" + # CPU case: pick the cheapest stoppable Linux SKU at runtime so the skill + # doesn't rot when SKUs change. Override by exporting VERIFY_STALE_CPU_TYPE. + CPU_TYPE=${VERIFY_STALE_CPU_TYPE:-$(brev search cpu --sort price --json \ + | jq -r '[.[] | select(.stoppable == true)] | .[0].type')} + [ -n "$CPU_TYPE" ] || { echo "ERROR: no stoppable CPU SKU available"; exit 1; } + brev create "$INSTANCE_NAME" --type "$CPU_TYPE" fi PROVISIONED_NEW=1 @@ -211,7 +271,9 @@ fi # Cleanup runs on success, error, and SIGINT. # Delete only what we provisioned. Reused boxes stay warm for next time. -trap '[ "$PROVISIONED_NEW" = "1" ] && brev delete "$INSTANCE_NAME" --yes || true' EXIT +# `brev delete` is non-interactive by default — there is no --yes flag, and passing one errors. +echo ">>> Brev instance: $INSTANCE_NAME (provisioned_new=$PROVISIONED_NEW; manual cleanup: brev delete $INSTANCE_NAME)" +trap '[ "$PROVISIONED_NEW" = "1" ] && brev delete "$INSTANCE_NAME" >/dev/null 2>&1 || true' EXIT ``` Wallclock cap per verification: **25 minutes** to accommodate two installs (reported version baseline + latest). If a provisioned box isn't ready in time, abort and treat as an infra failure (Step 11). @@ -260,7 +322,7 @@ The installer accepts the target ref via the `NEMOCLAW_INSTALL_TAG` env var (ver ```bash brev exec "$INSTANCE_NAME" "$RESET" -brev exec "$INSTANCE_NAME" "NEMOCLAW_INSTALL_TAG=$REPORTED_VERSION bash -c 'curl -fsSL https://nemoclaw.nvidia.com/install.sh | bash'" \ +brev exec "$INSTANCE_NAME" "NEMOCLAW_INSTALL_TAG=$REPORTED_VERSION bash -c 'curl -fsSL $INSTALL_URL | bash'" \ || BASELINE_INSTALL_FAILED=1 brev exec "$INSTANCE_NAME" "nemoclaw --version" ``` @@ -315,7 +377,7 @@ brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcri ```bash brev exec "$INSTANCE_NAME" "$RESET" -brev exec "$INSTANCE_NAME" "curl -fsSL https://nemoclaw.nvidia.com/install.sh | bash" +brev exec "$INSTANCE_NAME" "curl -fsSL $INSTALL_URL | bash" brev exec "$INSTANCE_NAME" "nemoclaw --version" brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh From 6cf97dcf7adcaf2cb4a6baa19d429be21186507f Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Wed, 6 May 2026 15:49:36 -0700 Subject: [PATCH 09/40] feat(verify-stale): detect behavior changed by design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Some bugs are filed against behavior that was deliberately removed or changed in a merged PR. Running the standard rubric on these produces a misleading verdict — the symptom "still reproduces" but the right answer is "won't fix, see PR #X." Issue #2791 is the prototype: `config set` was removed in #2227, the reporter tested a version that already had it gone, and the standard rubric would have buried that context under a low-confidence `verify-inconclusive` label. Add Step 8.5 with three independent detection signals: - Maintainer-attribution comment phrasing ("removed in #N", "by design", "wontfix", "intentional") from MEMBER/OWNER/COLLABORATOR authors. - A removal commit between reported version and `$LATEST` whose subject matches `\b(remove|delete|drop|deprecate)\b` and whose diff deletes the symbol implicated by the reproducer. - The implicated symbol absent from both reported version and `$LATEST`, meaning it was already gone when the issue was filed. When any signal fires, skip Step 9 scoring and Brev provisioning, apply a new `wontfix-by-design` label, and post a focused comment that links to the responsible PR. Detection on signals 2 and 3 can run as soon as the reported version is parsed, saving Brev cost on the entire class. Generalize the Step 3 idempotency marker regex from `v1` to `v\d+` so future skill versions can re-verify older-marked issues by tightening the regex to require a specific marker version. Add `wontfix-by-design` to the idempotency label list, the activity log entry options, the session summary, and the release sweep in `nemoclaw-maintainer-cut-release-tag`. Update the skill's frontmatter description to mention the new label and the local-first short-circuit. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../SKILL.md | 4 +- .../nemoclaw-maintainer-verify-stale/SKILL.md | 55 +++++++++++++++++-- 2 files changed, 52 insertions(+), 7 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md index aae53b1a806..74e645d9068 100644 --- a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md @@ -129,10 +129,10 @@ Confirm both tags point to the same commit on the remote. ## Step 7: Sweep Stale-Issue Verification Labels -Strip `fixed-on-latest` and `verify-inconclusive` from all open issues so the next `nemoclaw-maintainer-verify-stale` run re-evaluates against the new release. Without this sweep, "latest" drifts and verifications go silently stale. +Strip `fixed-on-latest`, `verify-inconclusive`, and `wontfix-by-design` from all open issues so the next `nemoclaw-maintainer-verify-stale` run re-evaluates against the new release. Without this sweep, "latest" drifts and verifications go silently stale. ```bash -for label in fixed-on-latest verify-inconclusive; do +for label in fixed-on-latest verify-inconclusive wontfix-by-design; do gh issue list --repo NVIDIA/NemoClaw --state open --label "$label" \ --json number -q '.[].number' \ | xargs -I{} gh issue edit {} --repo NVIDIA/NemoClaw --remove-label "$label" diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 3a30037291d..bb3f6a92a29 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -1,6 +1,6 @@ --- name: nemoclaw-maintainer-verify-stale -description: Verify whether old NVIDIA/NemoClaw bug reports still reproduce against the latest release. Picks candidate issues opened against older versions, reuses or provisions a Brev Linux box (CPU or GPU), attempts reproduction, scores confidence, and posts an evidence-backed comment with a label (fixed-on-latest or verify-inconclusive). Tag-only — never auto-closes. Linux-only in v1; Windows, macOS, and integration-token-dependent issues are skipped. Trigger keywords - verify stale, verify fixed, reproduce on latest, stale issue, old bug, fixed-on-latest, verify-inconclusive, drain backlog, brev verify. +description: Verify whether old NVIDIA/NemoClaw bug reports still reproduce against the latest release. Picks candidate issues opened against older versions, runs the reproducer locally first when possible, otherwise reuses or provisions a Brev Linux box (CPU or GPU), detects behavior that was intentionally changed, scores confidence, and posts an evidence-backed comment with a label (fixed-on-latest, wontfix-by-design, or verify-inconclusive). Tag-only — never auto-closes. Linux-only in v1; Windows, macOS, and integration-token-dependent issues are skipped. Trigger keywords - verify stale, verify fixed, reproduce on latest, stale issue, old bug, fixed-on-latest, wontfix-by-design, verify-inconclusive, drain backlog, brev verify. user_invocable: true --- @@ -72,8 +72,8 @@ Apply these rules in order. Drop any issue that fails a rule. **Idempotency:** drop if **either** of these is true: -- The issue carries a `fixed-on-latest` or `verify-inconclusive` label. (Cleared by the release sweep in `nemoclaw-maintainer-cut-release-tag` so the issue re-opens on each release.) -- A `<!-- nemoclaw-verify-stale v1 YYYY-MM-DD -->` comment was posted **within the last 7 days**. The marker carries a date so the candidate filter can apply a TTL — useful for the still-reproduces case (Step 9), where no label is applied and we want next week's run to re-verify rather than skip forever. +- The issue carries a `fixed-on-latest`, `verify-inconclusive`, or `wontfix-by-design` label. (Cleared by the release sweep in `nemoclaw-maintainer-cut-release-tag` so the issue re-opens on each release.) +- A comment matching `<!-- nemoclaw-verify-stale v\d+ YYYY-MM-DD -->` was posted **within the last 7 days**. The regex matches any marker version (`v1`, `v2`, …) so future skill versions can re-verify older-marked issues by tightening the regex (e.g. require a specific marker version). The marker carries a date so the candidate filter can apply a TTL — useful for the still-reproduces case (Step 9), where no label is applied and we want next week's run to re-verify rather than skip forever. **Candidate rule:** keep the issue if **either**: @@ -396,6 +396,49 @@ brev shell "$INSTANCE_NAME" --- +## Step 8.5: Detect "Behavior Changed by Design" + +Before scoring, check whether the symptom is intentional. Some bugs are filed against behavior that was **deliberately changed or removed** in a merged PR — running the rubric on these produces misleading verdicts. The symptom "still reproduces" but the right answer is "won't fix, see PR #X." Issue #2791 is the prototype: `config set` was removed in PR #2227, the reporter tested a version that already had it gone, and a verify-stale run that scored against the standard rubric would post a low-confidence `verify-inconclusive` label that buries the actual answer. + +**Signals that fire this branch (any one is sufficient):** + +1. **Maintainer attribution in comments.** Any comment by an author with `authorAssociation` of `MEMBER`, `OWNER`, or `COLLABORATOR` matches `removed in #\d+`, `removed in [Pp][Rr] ?#\d+`, `by design`, `wontfix`, `won't fix`, `not a bug`, or `intentional`. +2. **Removal commit in range.** A commit between the reported version and `$LATEST` has subject matching `(?i)\b(remove|delete|drop|deprecate)\b` AND its diff deletes the symbol implicated by the reproducer (CLI subcommand, function, flag). +3. **Symbol absent in both versions.** The implicated symbol (e.g. `config set`) is not present in either the reported version's source tree or `$LATEST`'s — meaning it was already gone when the issue was filed. + +**If any signal fires:** + +- **Skip the Step 9 score table** entirely. The "exit 0 + expected output" axis doesn't apply when the expected output is no longer the contract. +- **Skip Brev provisioning** if the signal fires before Step 7 — a remote run would just confirm what static analysis already proved. (Detection on signals 2 and 3 can run as soon as the reported version is parsed in Step 4.) +- **Apply label `wontfix-by-design`** (create the label if it doesn't exist; coordinate with maintainers on the canonical name before first use). +- **Use the by-design comment template below** instead of the standard Step 10 template. +- **@-mention the reporter** so they can object if the framing is wrong. +- **Never auto-close.** A maintainer pulls the trigger, same as the other label paths. + +**Comment template** (4-backtick outer fence so any nested code blocks render correctly): + +````markdown +## Stale-issue verification — behavior is by-design + +**Reported on:** v0.0.31 +**Verified on:** <tag-or-build-id> +**Outcome:** symptom reproduces, but the implicated behavior was intentionally changed. + +**Reference:** #<PR> (merged YYYY-MM-DD) <removed | renamed | replaced> the +`<symbol>` <command | function | flag>. <One-sentence summary of the new +intended workflow.> + +@<reporter> — recommend closing as "won't fix / by design". If the secondary +UX issue (e.g. confusing error message when invoking the removed symbol) is +worth tracking, that should be a fresh issue. + +<!-- nemoclaw-verify-stale v1 YYYY-MM-DD --> +```` + +**If no signal fires:** continue to Step 9 normally. + +--- + ## Step 9: Score Confidence Start at 0. Apply each rule that fires. @@ -635,7 +678,7 @@ After each issue (verified, inconclusive, or infra-failed), append to `~/develop **Latest install:** succeeded | failed (infra error) **Latest result:** not-reproduced (clean) | still-reproduces | partial / flake | n/a (skipped 8d) **Confidence:** 88 / 100 | n/a (still-reproduces) -**Label applied:** fixed-on-latest | verify-inconclusive | none (still-reproduces) | none (infra) +**Label applied:** fixed-on-latest | verify-inconclusive | wontfix-by-design | none (still-reproduces) | none (infra) **Brev wall time (approx):** N min --- @@ -658,7 +701,9 @@ At end of a batch session, prepend a session summary: ## YYYY-MM-DD — Verify Session **Issues considered:** N **Verified `fixed-on-latest`:** N +**Marked `wontfix-by-design`:** N **Marked `verify-inconclusive`:** N +**Local-first short-circuits (no Brev cost):** N **Skipped (Windows / macOS / integration / no version):** N **Infra failures:** N **Brev wall time:** N min · approx $X.XX @@ -689,4 +734,4 @@ Never stage or commit the log to the NemoClaw repo. ## Companion Behavior -`nemoclaw-maintainer-cut-release-tag` sweeps `fixed-on-latest` and `verify-inconclusive` from all open issues at release time. Without that sweep, "latest" drifts and verifications go stale silently. +`nemoclaw-maintainer-cut-release-tag` sweeps `fixed-on-latest`, `verify-inconclusive`, and `wontfix-by-design` from all open issues at release time. Without that sweep, "latest" drifts and verifications go stale silently. From 9fdcb25637db1597702e25d3a692f1abbc98808d Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Wed, 6 May 2026 15:51:27 -0700 Subject: [PATCH 10/40] fix(verify-stale): polish html extraction, tighten gpu keywords, env-var log path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The issue body that comes back from `gh issue view` for NV QA bugs is HTML, not markdown — `<pre>...</pre>` blocks and tables, with HTML-encoded entities. The previous Step 6 only mentioned `<pre>` in prose without an extractor that actually parsed it. Add a Python extractor that handles markdown fences (triple-backtick and tilde), HTML `<pre>`, and entity unescape in one pass, so verbatim reproducers from QA-filed issues land cleanly instead of falling through to LLM synthesis with a -30 penalty. Step 10 already had a comprehensive redaction regex table, but the patterns assumed plain text — tokens nested in HTML tags or attributes slipped through unredacted. Add an HTML to text pre-pass for issue-body excerpts before the regex table runs, so quoted excerpts from QA bodies get the same redaction coverage transcripts already do. Step 5's GPU keyword list flagged `inference` and `model serving`, both of which false-positive on CPU bugs (`models.providers.inference.baseUrl` is a config path, not a GPU need). Tighten to whole-word matches and swap in `vllm`, `tensorrt`, `L40S`, `L4`, `T4`. Step 12's activity log was hard-coded to one maintainer's personal organizer path. Read from `VERIFY_STALE_LOG_DIR` with the existing path as default, and require the skill to create the directory rather than assuming it exists, so the skill works in CI / shared-volume / other environments. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 43 +++++++++++++++++-- 1 file changed, 40 insertions(+), 3 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index bb3f6a92a29..e4fbbd4b282 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -141,7 +141,7 @@ Two real failure modes surfaced during the v1 dry-run. Test both before trusting **CPU vs GPU:** GPU if any of these signals are present, else CPU. - Labels: `Platform: GB10`, `Platform: DGX Spark`. -- Body keywords: `cuda`, `nvidia-smi`, `inference`, `model serving`, `H100`, `A100`, `GB10`, `DGX`. +- Body keywords (whole-word, case-insensitive): `nvidia-smi`, `cuda`, `H100`, `A100`, `L40S`, `L4`, `T4`, `GB10`, `DGX`, `vllm`, `tensorrt`. Match as whole words — `inference` and `model serving` are too noisy (e.g. `models.providers.inference.baseUrl` is a config path on CPU bugs, not a GPU need) and intentionally excluded. CPU default keeps cost low. Only escalate to GPU when the reproducer needs one. @@ -151,9 +151,30 @@ CPU default keeps cost low. Only escalate to GPU when the reproducer needs one. Extract whatever's available from the issue body. The decision about *whether the reproducer is good enough* lives in Step 8 (validate-on-baseline), not here. -1. **Verbatim:** the first fenced code block (triple-backtick or `<pre>`) containing a `nemoclaw` invocation. Save to `./reproducer.sh`. No confidence penalty (yet). +NV QA files most bugs through an HTML form, so issue bodies are typically a mix of `<pre>...</pre>` blocks and tables — not markdown fenced code blocks. Extraction must handle both shapes. + +1. **Verbatim:** the first markdown fence (```` ``` ```` or ```` ~~~ ````) **or** HTML `<pre>` block containing a `nemoclaw` invocation. Strip surrounding tags and unescape HTML entities before saving to `./reproducer.sh`. No confidence penalty (yet). 2. **No verbatim block found:** leave `./reproducer.sh` absent. Step 8b will synthesize from the issue body on demand and apply the **−30 synth penalty** at that point. +A robust extractor handles both shapes with the body fetched as JSON: + +```bash +BODY=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json body -q .body) + +REPRODUCER=$(printf '%s' "$BODY" | python3 -c ' +import re, sys, html +b = sys.stdin.read() +m = re.search(r"```(?:bash|sh)?\n(.*?nemoclaw.*?)\n```", b, re.S) +if not m: m = re.search(r"~~~(?:bash|sh)?\n(.*?nemoclaw.*?)\n~~~", b, re.S) +if not m: m = re.search(r"<pre[^>]*>(.*?nemoclaw.*?)</pre>", b, re.S) +if m: + text = re.sub(r"<[^>]+>", "", m.group(1)) + print(html.unescape(text).strip()) +') + +[ -n "$REPRODUCER" ] && printf '%s\n' "$REPRODUCER" > ./reproducer.sh +``` + The "give up immediately" path is gone. Synthesis happens at validation time so it has the baseline transcript to react to, not just the issue body in isolation. The give-up decision now lands in Step 8c when synth fails to produce a script that actually exposes the bug. --- @@ -522,6 +543,22 @@ The skill **never closes issues** in any branch. A maintainer pulls that trigger **Redaction pass before posting.** Run on **every** chunk of text quoted in the comment — issue body excerpts, baseline transcript, latest transcript, synth-repro scripts. Replace each match with `[REDACTED]`. The transcripts especially leak — they include full stdout/stderr from real installs and runs. +**HTML → text pre-pass for issue body excerpts.** NV QA bodies are HTML; tokens nested in `<pre>` tags or HTML attributes (e.g. `<a href="https://user:tok@host/...">`) slip past the regex patterns below if the input still has tags. Convert to plain text first, then redact: + +```bash +TEXT=$(printf '%s' "$BODY_EXCERPT" | python3 -c ' +import html, re, sys +b = sys.stdin.read() +b = re.sub(r"<br\s*/?>", "\n", b) +b = re.sub(r"</?(p|div|tr|td|th|li|pre)[^>]*>", "\n", b) +b = re.sub(r"<[^>]+>", "", b) +print(html.unescape(b)) +') +# Now apply the regex table below to $TEXT. +``` + +Transcripts and synth-repro scripts are already plain text and skip the pre-pass. + | Pattern | Targets | |---|---| | `(?i)(token\|secret\|password\|api[_-]?key\|bearer)[^\n]*[:=][^\n]*` | Inline credentials in env/config/log output | @@ -664,7 +701,7 @@ This degradation is expected — old releases rot. We still want to extract what ## Step 12: Log to Activity -After each issue (verified, inconclusive, or infra-failed), append to `~/development/daily-rhythm/activity/nemoclaw-verify-stale-log.md`. +After each issue (verified, inconclusive, by-design, or infra-failed), append to `${VERIFY_STALE_LOG_DIR:-$HOME/development/daily-rhythm/activity}/nemoclaw-verify-stale-log.md`. The default path matches the personal-organizer convention; export `VERIFY_STALE_LOG_DIR` to point elsewhere (CI, shared volume, etc.). Create the directory if missing — do not assume it exists. ```markdown ### NVIDIA/NemoClaw#<number> — <title> From e2039a8efcb1149cbd2f26ba08c4d2b81dc740b7 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Wed, 6 May 2026 16:07:51 -0700 Subject: [PATCH 11/40] fix(verify-stale): correct path map, openclaw reset, and other v1 polish MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Path-extraction map in Step 9 was written from assumption rather than verification — `src/lib/sandbox/`, `src/lib/openshell/`, and `src/lib/policy/` don't exist in the current repo, and OpenShell lives in a separate repo entirely. The +25 commits-touched-component signal would have silently misfired on every Sandbox / OpenShell / policy issue, underscoring real fixes. Replace with paths verified against the current tree (`nemoclaw/src/blueprint/`, `nemoclaw-blueprint/`, `nemoclaw/src/commands/`, etc.) and explicitly note the cross-repo OpenShell case is out of scope for v1. Step 8 reset wiped `~/.nemoclaw` but not `~/.openclaw`. Sandbox state has been writable-by-default under `.openclaw` since #2227, so it persists across the baseline → latest reinstall and contaminates the latest run. Add it to the reset. Anchor the `pkill -9 -f nemoclaw|openshell` patterns to a leading slash (`/nemoclaw`, `/openshell`) so the kill matches actual installed paths but not unrelated processes that mention these strings — including the agent harness running this skill if its working directory contains the word. Reorder the Step 10 redaction table to match the stated "longest, most specific patterns first" rule. The previous order had the generic `(token|secret|password|...)` pattern executing before JWT/AWS/NVIDIA patterns, which would have masked specific-token redactions with generic ones — losing the signal of *what kind* of credential leaked. Replace `git ls-remote git@github.com:NVIDIA/...` (Steps 2 and 4) with `gh api repos/NVIDIA/NemoClaw/tags --paginate --jq '.[].name'`. The SSH path required keys to be configured; `gh api` reuses whatever auth the user already has. Add a CLI-deps precondition check (`command -v gh brev jq python3 curl`) to Step 6.5 so the skill fails fast if any required dependency is missing, instead of failing late and confusingly mid-run. Step 11's keep-box-on-inconclusive said "delay cleanup by 30 minutes" but had no implementation. A backgrounded `sleep && brev delete` doesn't survive session end. Replace with skip-the-trap-entirely plus an explicit manual-cleanup reminder printed to the run output. Out-of-Scope was contradicting itself on macOS — Step 6.7 explicitly allows macOS for local-first runs. Clarify: Brev path is Linux-only, the local-first short-circuit on a maintainer's laptop works on macOS for manual single-issue runs. Add a `local (no Brev — Step 6.7 short-circuit)` option to the Step 12 log entry's Box field so local-first runs round-trip the activity log correctly. Frontmatter description said "latest release" but NemoClaw uses tags, not GitHub Releases — fix to "latest tag" to match Step 2. Step 4's implementer note said "Two real failure modes" but listed three; fix the off-by-one. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 88 ++++++++++--------- 1 file changed, 48 insertions(+), 40 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index e4fbbd4b282..edb583009aa 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -1,6 +1,6 @@ --- name: nemoclaw-maintainer-verify-stale -description: Verify whether old NVIDIA/NemoClaw bug reports still reproduce against the latest release. Picks candidate issues opened against older versions, runs the reproducer locally first when possible, otherwise reuses or provisions a Brev Linux box (CPU or GPU), detects behavior that was intentionally changed, scores confidence, and posts an evidence-backed comment with a label (fixed-on-latest, wontfix-by-design, or verify-inconclusive). Tag-only — never auto-closes. Linux-only in v1; Windows, macOS, and integration-token-dependent issues are skipped. Trigger keywords - verify stale, verify fixed, reproduce on latest, stale issue, old bug, fixed-on-latest, wontfix-by-design, verify-inconclusive, drain backlog, brev verify. +description: Verify whether old NVIDIA/NemoClaw bug reports still reproduce against the latest tag. Picks candidate issues opened against older versions, runs the reproducer locally first when possible (Linux or macOS), otherwise reuses or provisions a Brev Linux box (CPU or GPU), detects behavior that was intentionally changed, scores confidence, and posts an evidence-backed comment with a label (fixed-on-latest, wontfix-by-design, or verify-inconclusive). Tag-only — never auto-closes. Brev verification is Linux-only in v1; Windows and integration-token-dependent issues are skipped. Trigger keywords - verify stale, verify fixed, reproduce on latest, stale issue, old bug, fixed-on-latest, wontfix-by-design, verify-inconclusive, drain backlog, brev verify. user_invocable: true --- @@ -38,19 +38,18 @@ In batch mode, work through items one at a time. Present each verification plan ## Step 2: Detect the Latest NemoClaw Version -Try GitHub releases first; fall back to the highest semver git tag if no release is published. NemoClaw currently tags but does not publish releases, so the fallback is the load-bearing path today. +Try GitHub releases first; fall back to the highest semver tag from the GitHub API if no release is published. NemoClaw currently tags but does not publish releases, so the fallback is the load-bearing path today. Use `gh api` rather than `git ls-remote` so the skill works regardless of SSH key setup, and reuses the auth `gh` already has. ```bash LATEST=$(gh release view --repo NVIDIA/NemoClaw --json tagName -q .tagName 2>/dev/null) if [ -z "$LATEST" ]; then - LATEST=$(git ls-remote --tags --refs git@github.com:NVIDIA/NemoClaw.git \ - | awk -F/ '{print $NF}' \ + LATEST=$(gh api repos/NVIDIA/NemoClaw/tags --paginate --jq '.[].name' \ | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' \ | sort -V | tail -1) fi -echo "Latest release: $LATEST" +echo "Latest tag: $LATEST" ``` This is the version the skill will verify against. Record it — every comment must cite it. @@ -104,8 +103,7 @@ Collect every match from sources 2 and 3 (a single body may mention multiple ver - Versions parsed from prose that happen to look semver-ish but aren't releases. ```bash -git ls-remote --tags --refs git@github.com:NVIDIA/NemoClaw.git \ - | awk -F/ '{print $NF}' > /tmp/nemoclaw-tags.txt +gh api repos/NVIDIA/NemoClaw/tags --paginate --jq '.[].name' > /tmp/nemoclaw-tags.txt # For each candidate version V: grep -Fxq "$V" /tmp/nemoclaw-tags.txt || drop_version "$V" @@ -119,7 +117,7 @@ If no version survives, drop the issue from the candidate set — we cannot esta ### Implementer note: regex-pipeline pitfalls -Two real failure modes surfaced during the v1 dry-run. Test both before trusting your implementation: +Three real failure modes surfaced during the v1 dry-run. Test each before trusting your implementation: 1. **Empty-match handling.** A naive pipeline like `[scan(regex)] | first | .[0] | tonumber // fallback` silently dropped 9 real candidates (e.g. #2861 with `NemoClaw 0.0.32`, #2604 with `NemoClaw: 0.0.28`). When `scan` returns no matches, `[]` flows in, `first` returns null, `null | .[0]` errors, and `//` does not propagate cleanly through the error. Bind each pass to a named variable, coalesce at the end: @@ -181,9 +179,14 @@ The "give up immediately" path is gone. Synthesis happens at validation time so ## Step 6.5: Verify Preconditions -Confirm `brev` is authenticated and the install URL resolves before paying any cost. Credentials live in `~/.brev/credentials.json` and are reused across shells under the same OS user, so once authenticated the auth check is a no-op until the token expires. +Confirm CLI dependencies are available, `brev` is authenticated, and the install URL resolves before paying any cost. Credentials live in `~/.brev/credentials.json` and are reused across shells under the same OS user, so once authenticated the auth check is a no-op until the token expires. ```bash +# CLI deps — fail fast if anything later in the skill needs them but they're missing. +for cmd in gh brev jq python3 curl; do + command -v "$cmd" >/dev/null 2>&1 || { echo "ERROR: missing required dependency: $cmd"; exit 1; } +done + # Brev auth — short-circuit only after the auth check, not before. brev ls --json >/dev/null 2>&1 || { echo "Brev not authenticated. Choose one:" @@ -319,11 +322,16 @@ NemoClaw spawns OpenShell sandboxes (containers), runtime services, and listenin ```bash RESET=$(cat <<'SCRIPT' nemoclaw destroy --all --force 2>/dev/null || true -pkill -9 -f nemoclaw 2>/dev/null || true -pkill -9 -f openshell 2>/dev/null || true +# Anchor pkill patterns to "/nemoclaw" / "/openshell" path components so the kill doesn't +# match unrelated processes that happen to mention these strings (including the agent +# harness running this skill if its working dir contains the word). +pkill -9 -f '/nemoclaw([[:space:]]|$)' 2>/dev/null || true +pkill -9 -f '/openshell([[:space:]]|$)' 2>/dev/null || true docker ps -a --filter "name=openshell-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true docker ps -a --filter "name=nemoclaw-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true -rm -rf ~/.nemoclaw 2>/dev/null +# Sandbox state lives in ~/.openclaw (default-writable since #2227); ~/.nemoclaw holds CLI state. +# Wipe both so the latest install starts clean. +rm -rf ~/.nemoclaw ~/.openclaw 2>/dev/null sudo -n rm -f /usr/local/bin/nemoclaw 2>/dev/null || true sudo -n rm -rf /usr/local/lib/nemoclaw 2>/dev/null || true for port in 8080 18789 9119; do fuser -k -n tcp $port 2>/dev/null || true; done @@ -483,14 +491,14 @@ The skill needs to know *which* path to `git log v<reported>..$LATEST -- <path>` - `/usr/local/bin/nemoclaw*` → `bin/` - `~/.nemoclaw/<rel>` → most often runtime state, drop unless the bug is config-related → `src/lib/config/` - In-repo paths (e.g., `bin/lib/policies.js` mentioned literally) → use as-is -2. **Component-label-to-directory map.** Pick the first match: - - `NemoClaw CLI` → `bin/`, `src/` - - `Sandbox` → `src/lib/sandbox/`, `nemoclaw/sandbox/` - - `OpenShell` → `nemoclaw/openshell/`, `src/lib/openshell/` - - `Docker` → `Dockerfile`, `scripts/install-openshell.sh` - - `Getting Started` → `docs/`, `install.sh` - - `Integration: <X>` (when not in skip list) → `src/lib/integrations/<x>/` -3. **Title keywords.** "TUI" → `src/tui/`, "policy" → `src/lib/policy/`, "inference" → `src/lib/inference/`. +2. **Component-label-to-directory map.** Pick the first match. Paths verified against the current repo layout — drop any path that doesn't exist on the tag at `$LATEST` rather than passing it to `git log`. + - `NemoClaw CLI` → `bin/`, `src/lib/`, `nemoclaw/src/commands/` + - `Sandbox` → `nemoclaw/src/blueprint/`, `nemoclaw-blueprint/` + - `OpenShell` → cross-repo (lives at `github.com/NVIDIA/OpenShell`, not in this repo). Skip the +25 signal for OpenShell-only issues; cross-repo `git log` is out of v1 scope. + - `Docker` → `Dockerfile`, `Dockerfile.base`, `scripts/install-openshell.sh`, `scripts/install.sh` + - `Getting Started` → `docs/`, `scripts/install.sh` + - `Integration: <X>` — no `src/lib/integrations/` exists in this repo. Skip the +25 signal for integration-component issues unless source 1 (file paths in body) yielded a path. +3. **Title keywords.** "policy" → `nemoclaw-blueprint/policies/`, `nemoclaw/src/blueprint/`. "inference" → `docs/inference/` is docs-only; skip the +25 signal unless source 1 surfaces actual code paths. If none of the above produces a path, **skip the +25 signal entirely** rather than guessing. Floating the +25 on every issue would inflate scores meaninglessly. @@ -559,23 +567,23 @@ print(html.unescape(b)) Transcripts and synth-repro scripts are already plain text and skip the pre-pass. -| Pattern | Targets | -|---|---| -| `(?i)(token\|secret\|password\|api[_-]?key\|bearer)[^\n]*[:=][^\n]*` | Inline credentials in env/config/log output | -| `(?i)authorization:\s*\S+` | HTTP auth headers (often Bearer + JWT) | -| `eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}` | JWT tokens | -| `gh[pousr]_[A-Za-z0-9]{36,}` | GitHub PATs / install tokens | -| `AKIA[0-9A-Z]{16}` | AWS access key IDs | -| `(?i)aws_secret_access_key\s*=\s*\S+` | AWS secret keys | -| `(?i)nvapi-[A-Za-z0-9_-]{20,}` | NVIDIA API keys (NIM / build.nvidia.com) | -| URLs containing `@` before the host (e.g., `https://user:pw@host/...`) | Basic-auth credentials in URLs | -| `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}` | Email addresses (PII) | -| `\b[A-Za-z0-9+/]{60,}={0,2}\b` | Long base64 blobs (likely keys/sessions; tune length to taste — too short hits legit data) | -| `\b\w+\.(nvidia\.internal\|nv-internal\.com\|nvidia\.dev)\b` | Internal hostnames (extend list per team) | - -**File paths under the reporter's home directory** (`/Users/<name>/`, `/home/<name>/`) → replace with `~/`. Catches incidental username PII. - -**Order matters.** Run the longest, most-specific patterns first (JWT, AWS, NVIDIA-API) before the generic base64 catchall, otherwise the catchall masks the specific match and you lose the fact that *what* was redacted was a JWT vs a session blob. +**Order matters and the table below is in execution order.** Longest, most-specific patterns first; generic catchalls last. Otherwise the catchall masks specific matches and you lose track of what was actually redacted (JWT vs session blob vs random base64). + +| # | Pattern | Targets | +|---|---|---| +| 1 | `eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}` | JWT tokens | +| 2 | `gh[pousr]_[A-Za-z0-9]{36,}` | GitHub PATs / install tokens | +| 3 | `(?i)nvapi-[A-Za-z0-9_-]{20,}` | NVIDIA API keys (NIM / build.nvidia.com) | +| 4 | `AKIA[0-9A-Z]{16}` | AWS access key IDs | +| 5 | `(?i)aws_secret_access_key\s*=\s*\S+` | AWS secret keys | +| 6 | `(?i)authorization:\s*\S+` | HTTP auth headers (often Bearer + JWT) | +| 7 | URLs containing `@` before the host (e.g., `https://user:pw@host/...`) | Basic-auth credentials in URLs | +| 8 | `(?i)(token\|secret\|password\|api[_-]?key\|bearer)[^\n]*[:=][^\n]*` | Inline credentials in env/config/log output | +| 9 | `\b\w+\.(nvidia\.internal\|nv-internal\.com\|nvidia\.dev)\b` | Internal hostnames (extend list per team) | +| 10 | `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}` | Email addresses (PII) | +| 11 | `\b[A-Za-z0-9+/]{60,}={0,2}\b` | Long base64 blobs (likely keys/sessions; tune length to taste — too short hits legit data) | + +**File paths under the reporter's home directory** (`/Users/<name>/`, `/home/<name>/`) → replace with `~/`. Run last; catches incidental username PII. **Comment template (fixed / inconclusive — bug not reproduced on latest):** @@ -695,7 +703,7 @@ The next weekly run retries naturally. This degradation is expected — old releases rot. We still want to extract whatever signal we can from the latest run plus PR/commit evidence, just at a more conservative confidence ceiling. -**Keep-box-on-inconclusive.** When `verify-inconclusive` lands (Step 8c gave up, or Step 9 score < 60), **delay the cleanup `brev delete` by 30 minutes** if the box was provisioned by this run. Print the `brev shell "$INSTANCE_NAME"` command in the run output so a maintainer can hop in and triage. Reused boxes stay regardless. Ship-failed verifications are the exact case where having an inspectable artifact pays for itself. +**Keep-box-on-inconclusive.** When `verify-inconclusive` lands (Step 8c gave up, or Step 9 score < 60), **skip the cleanup trap** for this run if the box was provisioned by this run — set `PROVISIONED_NEW=0` before the trap fires so the EXIT handler is a no-op. Print the `brev shell "$INSTANCE_NAME"` command and an explicit `brev delete "$INSTANCE_NAME"` reminder in the run output so the maintainer can triage and clean up manually. Reused boxes stay regardless. Ship-failed verifications are the exact case where having an inspectable artifact pays for itself; an unbounded sleep-and-delete in the background isn't reliable across session ends, so we leave deletion explicit. --- @@ -709,7 +717,7 @@ After each issue (verified, inconclusive, by-design, or infra-failed), append to **Reported on:** v0.0.31 **Verified on:** v0.0.34 **Environment:** CPU | GPU (<instance type>) -**Box:** reused <name> | provisioned <name> +**Box:** reused <name> | provisioned <name> | local (no Brev — Step 6.7 short-circuit) **Baseline install:** succeeded | failed (degraded mode) **Baseline match:** validated (verbatim) | validated (synth) | failed (verify-inconclusive) | skipped **Latest install:** succeeded | failed (infra error) @@ -762,7 +770,7 @@ Never stage or commit the log to the NemoClaw repo. ## Out of Scope (v1) - Auto-closing issues. Always tag-only; a human pulls the trigger. -- macOS verification. Brev offers no macOS instances and local-laptop runs are not unattended. +- macOS verification *via the Brev path*. Brev offers no macOS instances. The Step 6.7 local-first short-circuit *does* run on a maintainer's macOS laptop — so manual single-issue runs against pure-CLI bugs work on macOS. The weekly batch cron is Linux-only because that path always uses Brev. - Issues requiring third-party integration credentials (Slack, Discord, Telegram, Hermes, OpenClaw, WeChat). - Service-account bot identity. v1 runs under each maintainer's own GitHub credentials. - Versioned labels. A single `fixed-on-latest` label is swept on each release cut. From e7be9f8d3415104255c5fc70c4480858af5293d6 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Wed, 6 May 2026 16:43:02 -0700 Subject: [PATCH 12/40] fix(verify-stale): tighten by-design detection with evidence and self-verification MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 8.5 was producing comments that were correct in direction but hand-wavy on substance. A side-by-side against an independent analysis of #2168 surfaced the gap: that analysis cited specific file:line locations, separated the literal bug-as-filed from a related failure mode that still exists on latest, called out existing CI coverage of the new workflow, and self-corrected an overstatement when prompted to re-verify. Today the skill required none of that. Restructure Step 8.5 into substeps so the rigor is mechanical, not prompt-dependent: - 8.5a: signal detection now requires verifiable evidence per signal — comment URL + quoted phrase (signal 1), commit SHA + diff range (signal 2), grep commands + outputs (signal 3). Add a sub-case for vestigial deprecation shims so a stub doesn't silently fail signal 3 by appearing in latest. - 8.5b: pre-check for related failure modes. Grep latest for the bug's symptom keywords (not the removed symbol) and require the comment to call out anything that surfaces. This is the section the side-by-side identified as missing — saying "the bug as filed can't reproduce" is not the same as "every bug shaped like this is fixed." - 8.5c: check existing test coverage for the new workflow and cite up to three test paths if found. Strengthens the comment from "trust me, it was removed" to "the new workflow is exercised by these tests." - 8.5d: explicit self-verification pass — re-run every cited command before composing the comment; bail to verify-inconclusive on any discrepancy. LLMs confidently overstate; mechanical re-verification catches it without needing a human prompt. Replace the by-design comment template with one that has mandatory sections matching the substeps: "What's structurally fixed", "Vestigial references", "What's not literally the same bug", "Existing CI coverage", "Recommendation". Each section either has concrete content or is explicitly omitted — no hand-wavy claims slip through. Add NVBugs cross-reference extraction to Step 4 (`grep -oE '\[NVB#[0-9]+\]'`) so the by-design template can append the standard "NVBugs#NNNNNNN will need a separate update; closing this GitHub issue won't propagate" reminder when the issue body carries that footer (most NV QA bugs do). Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 135 +++++++++++++++--- 1 file changed, 117 insertions(+), 18 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index edb583009aa..9bba80a2eca 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -115,6 +115,14 @@ If no version survives, drop the issue from the candidate set — we cannot esta **Variable format for downstream steps.** Set `REPORTED_VERSION` to the **full tag string** (e.g., `REPORTED_VERSION="v0.0.32"`), not just the patch number. Step 8a's installer expects the full tag via the `NEMOCLAW_INSTALL_TAG` env var. +**NVBugs cross-reference.** Many NV QA bugs include an NVBugs ticket footer like `[NVB#6100043]`. Extract it at the same time as the version so Step 8.5's comment template (and any other comment template that wants to mention it) can include the cross-reference: + +```bash +NVBUGS_REF=$(printf '%s' "$BODY" | grep -oE '\[NVB#[0-9]+\]' | head -1) +``` + +Templates ignore this when empty. When present, the comment must note that closing the GitHub issue does not propagate to NVBugs and QA needs to update the ticket separately. + ### Implementer note: regex-pipeline pitfalls Three real failure modes surfaced during the v1 dry-run. Test each before trusting your implementation: @@ -427,43 +435,134 @@ brev shell "$INSTANCE_NAME" ## Step 8.5: Detect "Behavior Changed by Design" -Before scoring, check whether the symptom is intentional. Some bugs are filed against behavior that was **deliberately changed or removed** in a merged PR — running the rubric on these produces misleading verdicts. The symptom "still reproduces" but the right answer is "won't fix, see PR #X." Issue #2791 is the prototype: `config set` was removed in PR #2227, the reporter tested a version that already had it gone, and a verify-stale run that scored against the standard rubric would post a low-confidence `verify-inconclusive` label that buries the actual answer. +Before scoring, check whether the symptom is intentional. Some bugs are filed against behavior that was **deliberately changed or removed** in a merged PR — running the standard rubric on these produces misleading verdicts. The symptom "still reproduces" but the right answer is "won't fix, see PR #X." Issue #2791 is the prototype: `config set` was removed in PR #2227, the reporter tested a version that already had it gone, and a standard rubric run would have buried that context under a low-confidence `verify-inconclusive` label. + +This step is split into substeps so the rigor is mechanical, not optional. Every claim in the final comment must be backed by a verifiable evidence block — a comment URL with quoted phrase, a commit SHA with diff range, or a grep command with its actual output. Hand-wavy claims fail Step 8.5d's self-verification pass and force a bail to `verify-inconclusive`. + +### Step 8.5a: Run signal detection + +Any single signal is sufficient to trigger the by-design branch. + +**Signal 1 — Maintainer attribution in comments.** Any comment by an author with `authorAssociation` of `MEMBER`, `OWNER`, or `COLLABORATOR` matches `removed in #\d+`, `removed in [Pp][Rr] ?#\d+`, `by design`, `wontfix`, `won't fix`, `not a bug`, or `intentional`. + +```bash +gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ + --jq '.comments[] + | select(.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR") + | select(.body | test("removed in #\\d+|by design|wontfix|won.t fix|not a bug|intentional"; "i")) + | {url, author: .author.login, body}' +``` + +Capture for evidence: comment URL + author login + the exact quoted phrase. + +**Signal 2 — Removal commit in range.** A commit between the reported version and `$LATEST` has subject matching `(?i)\b(remove|delete|drop|deprecate)\b` AND its diff deletes the symbol implicated by the reproducer (CLI subcommand, function, flag). -**Signals that fire this branch (any one is sufficient):** +```bash +# Find candidate removal commits. +git log "$REPORTED_VERSION".."$LATEST" --grep='remove\|delete\|drop\|deprecate' -i --oneline + +# For each candidate, confirm the diff actually deletes the symbol (not just renames). +git log -p <candidate-sha> -- src/ bin/ nemoclaw/src/ | grep -nE '^-.*\b<symbol>\b' +``` + +Capture for evidence: commit SHA + each `file:line` block of deletions touching the symbol. + +**Signal 3 — Symbol absent in both reported version and latest.** The implicated symbol (e.g. `config set`) is not present in either tag's source tree — meaning the responsible change landed before the version the reporter tested. This is the #2791 case. + +```bash +git grep -n "<symbol>" "$REPORTED_VERSION" -- src/ bin/ nemoclaw/ # expect: zero matches (or shim-only — see sub-case) +git grep -n "<symbol>" "$LATEST" -- src/ bin/ nemoclaw/ # expect: zero matches (or shim-only) +``` + +Capture for evidence: both grep commands and their (empty) outputs. + +**Sub-case for signals 2 and 3 — vestigial deprecation shims.** It's common for a removed symbol to survive in latest *only* as a deprecation message (e.g., a CLI subcommand that prints `"--<flag> was removed; use <X> instead"` and exits non-zero). When a grep returns matches in latest, inspect each `file:line`. If every match is a deprecation stub with no functional effect on the bug-as-filed, signal 2 or 3 still fires; record the shim locations and behavior as a separate evidence block. Do not silently treat shims as functional code, and do not silently treat them as absence. + +### Step 8.5b: Pre-check related failure modes -1. **Maintainer attribution in comments.** Any comment by an author with `authorAssociation` of `MEMBER`, `OWNER`, or `COLLABORATOR` matches `removed in #\d+`, `removed in [Pp][Rr] ?#\d+`, `by design`, `wontfix`, `won't fix`, `not a bug`, or `intentional`. -2. **Removal commit in range.** A commit between the reported version and `$LATEST` has subject matching `(?i)\b(remove|delete|drop|deprecate)\b` AND its diff deletes the symbol implicated by the reproducer (CLI subcommand, function, flag). -3. **Symbol absent in both versions.** The implicated symbol (e.g. `config set`) is not present in either the reported version's source tree or `$LATEST`'s — meaning it was already gone when the issue was filed. +A by-design verdict says "the bug *as filed* can't reproduce." It does NOT say "every bug shaped like this is fixed." Before drafting the comment, search latest's source for code paths that could still produce the issue's described **symptom** (not the literal removed flag/symbol — the symptom). + +```bash +# Use the issue's symptom keywords, not the removed symbol. +git grep -nE "<symptom-keyword-1>|<symptom-keyword-2>" "$LATEST" -- src/ nemoclaw/src/ +``` -**If any signal fires:** +For #2168 the literal flag is `--dangerously-skip-permissions`, but the symptom is "sandbox created but not registered in CLI." Grepping for `register.*[Ss]andbox`, the readiness-gate / cleanup-failure path in `src/lib/onboard.ts` surfaces as a related-but-different way to produce an orphan sandbox. + +If a related failure mode is found, the by-design comment MUST include a "What's not literally the same bug" section that names it with `file:line`. Don't suppress the call-out by claiming "the symptom is impossible" when the symptom can be reached via a different path. + +### Step 8.5c: Check existing test coverage + +Search the repo for tests that exercise the NEW intended workflow (the one that replaced the removed symbol). Citing them strengthens the comment from "trust me, it was removed" to "the new workflow is exercised by these tests." + +```bash +git grep -lnE "<new-workflow-keyword>" -- test/ nemoclaw/src/ 2>/dev/null | head -5 +``` + +Cite at most three concrete test paths. If none exist, omit the section — do not invent paths. + +### Step 8.5d: Self-verification pass before posting + +Re-run every grep / git / `gh` command cited in the evidence blocks before composing the comment. If any cited `file:line`, commit SHA, or quoted output doesn't reproduce on a fresh invocation, **stop and revise** — or bail to `verify-inconclusive` if the discrepancy can't be resolved. The cost of an incorrect "I checked and X is gone" claim in a public comment is higher than spending 30 seconds re-checking. This step exists because LLMs can confidently overstate; mechanical re-verification catches it. + +### Step 8.5e: If any signal fires - **Skip the Step 9 score table** entirely. The "exit 0 + expected output" axis doesn't apply when the expected output is no longer the contract. -- **Skip Brev provisioning** if the signal fires before Step 7 — a remote run would just confirm what static analysis already proved. (Detection on signals 2 and 3 can run as soon as the reported version is parsed in Step 4.) -- **Apply label `wontfix-by-design`** (create the label if it doesn't exist; coordinate with maintainers on the canonical name before first use). +- **Skip Brev provisioning** if the signal fires before Step 7 — a remote run would just confirm what static analysis already proved. (Signals 2 and 3 can run as soon as the reported version is parsed in Step 4.) +- **Apply label `wontfix-by-design`** (`gh label create wontfix-by-design ...` if it doesn't exist; coordinate with maintainers on the canonical name before first use). - **Use the by-design comment template below** instead of the standard Step 10 template. - **@-mention the reporter** so they can object if the framing is wrong. - **Never auto-close.** A maintainer pulls the trigger, same as the other label paths. -**Comment template** (4-backtick outer fence so any nested code blocks render correctly): +### By-design comment template + +Mandatory sections in this order. Omit only the sections explicitly noted as omittable. ````markdown ## Stale-issue verification — behavior is by-design -**Reported on:** v0.0.31 -**Verified on:** <tag-or-build-id> -**Outcome:** symptom reproduces, but the implicated behavior was intentionally changed. +**Reported on:** v0.0.<X> +**Verified on:** v0.0.<Y> (PR #<NNNN> first shipped in v0.0.<Z>) +**Outcome:** symptom reproduces against the reproducer as filed, but the implicated behavior was intentionally changed. + +### What's structurally fixed + +- `<file:line>` — `<one-sentence summary of the change at that location>` +- `<file:line>` — `<…>` -**Reference:** #<PR> (merged YYYY-MM-DD) <removed | renamed | replaced> the -`<symbol>` <command | function | flag>. <One-sentence summary of the new -intended workflow.> +The new workflow is `<one-sentence: how to do what the user was trying to do>`. -@<reporter> — recommend closing as "won't fix / by design". If the secondary -UX issue (e.g. confusing error message when invoking the removed symbol) is -worth tracking, that should be a fresh issue. +### Vestigial references + +- `<file:line>` — `<deprecation behavior: e.g. "prints '--<flag> was removed; use <X> instead' and exits 1; no functional effect">` + +(Omit this section entirely when the symbol is fully gone with no surviving stubs.) + +### What's not literally the same bug + +`<one-sentence acknowledgement of the related failure mode found in Step 8.5b, with file:line>` — OR — `None. The symptom requires the removed symbol; no related code path produces it on latest.` + +### Existing CI coverage + +- `<test/path/file>` — `<one-sentence: what this test demonstrates about the new workflow>` + +(Omit when no direct test exists. Do not invent paths.) + +### Recommendation + +@<reporter> — recommend closing as "won't fix / by design". If a related symptom (e.g. `<related failure mode from above>`) is hitting you on ≥ v0.0.<Z>, please file a fresh issue with a v0.0.<Z>+ reproducer. + +`<NVBugs cross-ref line — see below>` <!-- nemoclaw-verify-stale v1 YYYY-MM-DD --> ```` +**NVBugs cross-ref line.** If `NVBUGS_REF` was set in Step 4, append: + +> NVBugs<NVBUGS_REF without brackets> will need a separate update; closing this GitHub issue won't propagate. + +Otherwise omit the sentence. + **If no signal fires:** continue to Step 9 normally. --- From 32c61da3c7f918d823f121a964e4841287fa78ed Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Wed, 6 May 2026 16:44:23 -0700 Subject: [PATCH 13/40] fix(verify-stale): broaden signal 2 candidate search to pickaxe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous signal-2 candidate query narrowed by commit subject (`remove|delete|drop|deprecate`) before checking whether the diff deleted the implicated symbol. That filter excludes the common case where a removal lands inside a `refactor(...)` or `feat(...)` commit — e.g., PR #2227 removed `--dangerously-skip-permissions` under a "refactor(sandbox): default to mutable config" subject. A literal follower of the previous rule would have concluded signal 2 doesn't fire on issue #2168 even though the flag was deliberately removed. Switch the primary candidate lookup to `git log -S<symbol>` pickaxe search, which finds every commit whose diff changes the count of the symbol regardless of subject wording. Keep the subject-keyword query as a supplementary lookup for narrowing when the pickaxe returns many commits. Note explicitly that the commit's actual subject doesn't need to mention removal. Surfaced while testing the new Step 8.5 rigor against #2168. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 9bba80a2eca..dcb3a0ae044 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -455,17 +455,22 @@ gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ Capture for evidence: comment URL + author login + the exact quoted phrase. -**Signal 2 — Removal commit in range.** A commit between the reported version and `$LATEST` has subject matching `(?i)\b(remove|delete|drop|deprecate)\b` AND its diff deletes the symbol implicated by the reproducer (CLI subcommand, function, flag). +**Signal 2 — Removal commit in range.** A commit between the reported version and `$LATEST` deletes the symbol implicated by the reproducer (CLI subcommand, function, flag). The commit subject does NOT need to mention "remove" / "delete" — many removals ride into a `refactor(...)` or `feat(...)` commit (e.g. PR #2227 removed `--dangerously-skip-permissions` under a `refactor(sandbox): ...` subject). Use git's pickaxe to find the responsible commit by content: ```bash -# Find candidate removal commits. +# Pickaxe: list every commit whose diff changes the count of <symbol> occurrences. +# Reverse order so the earliest removal commit lands first in the list. +git log "$REPORTED_VERSION".."$LATEST" -S'<symbol>' --reverse --oneline -- src/ bin/ nemoclaw/src/ + +# Subject-keyword narrowing is only a SUPPLEMENTARY lookup — useful when the +# pickaxe returns many commits and you want to focus on the obviously-removal one. git log "$REPORTED_VERSION".."$LATEST" --grep='remove\|delete\|drop\|deprecate' -i --oneline -# For each candidate, confirm the diff actually deletes the symbol (not just renames). +# For each candidate, confirm the diff actually deletes the symbol (not just renames or moves it). git log -p <candidate-sha> -- src/ bin/ nemoclaw/src/ | grep -nE '^-.*\b<symbol>\b' ``` -Capture for evidence: commit SHA + each `file:line` block of deletions touching the symbol. +Capture for evidence: commit SHA + each `file:line` block of deletions touching the symbol. Note the commit's actual subject — don't assume it says "remove." **Signal 3 — Symbol absent in both reported version and latest.** The implicated symbol (e.g. `config set`) is not present in either tag's source tree — meaning the responsible change landed before the version the reporter tested. This is the #2791 case. From 9552b593a3237184ea0b4cda5fbb1824dfb87331 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Wed, 6 May 2026 16:54:16 -0700 Subject: [PATCH 14/40] fix(verify-stale): use wontfix label, add verification-mode + tag anchoring MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes that surfaced from running Step 8.5 against issue #2168. Use the existing repo `wontfix` label for the by-design path instead of creating a new `wontfix-by-design` label. The repo already has `wontfix` and it's already in Step 3's issue-type skip list, so applying it gives idempotency for free without proliferating labels. The by-design verdict is encoded in the marker comment, not the label vocabulary. Important consequence: the release sweep in `nemoclaw-maintainer-cut-release-tag` does NOT clear `wontfix` — that label is also applied for non-skill reasons (scope, priority, dup decisions made by maintainers), and sweeping it would erase human triage work. Sweep stays scoped to `fixed-on-latest` and `verify-inconclusive`, which are skill-only. Add a `**Verification mode:**` line to the by-design comment template that explicitly says "static analysis at the verified-on tag — no runtime reproduction." This was the honesty an independent side-by-side analysis of #2168 caught: the previous template implied runtime confirmation even though Step 8.5 short-circuits Brev provisioning by design. Add a tag-anchoring rule above the template too — every `file:line` cite must refer to the verified-on tag, not the maintainer's HEAD, since line numbers drift between tags and main and we want comments to stay reproducible by anyone reading them later. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-cut-release-tag/SKILL.md | 4 ++-- .../nemoclaw-maintainer-verify-stale/SKILL.md | 15 +++++++++------ 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md index 74e645d9068..118eec08dbc 100644 --- a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md @@ -129,10 +129,10 @@ Confirm both tags point to the same commit on the remote. ## Step 7: Sweep Stale-Issue Verification Labels -Strip `fixed-on-latest`, `verify-inconclusive`, and `wontfix-by-design` from all open issues so the next `nemoclaw-maintainer-verify-stale` run re-evaluates against the new release. Without this sweep, "latest" drifts and verifications go silently stale. +Strip `fixed-on-latest` and `verify-inconclusive` from all open issues so the next `nemoclaw-maintainer-verify-stale` run re-evaluates against the new release. Without this sweep, "latest" drifts and verifications go silently stale. The skill's by-design path uses the existing repo `wontfix` label, which is **not** swept here — `wontfix` is also applied for non-skill reasons (scope, priority, dup), so clearing it would erase human triage work. ```bash -for label in fixed-on-latest verify-inconclusive wontfix-by-design; do +for label in fixed-on-latest verify-inconclusive; do gh issue list --repo NVIDIA/NemoClaw --state open --label "$label" \ --json number -q '.[].number' \ | xargs -I{} gh issue edit {} --repo NVIDIA/NemoClaw --remove-label "$label" diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index dcb3a0ae044..48df8291e8d 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -1,6 +1,6 @@ --- name: nemoclaw-maintainer-verify-stale -description: Verify whether old NVIDIA/NemoClaw bug reports still reproduce against the latest tag. Picks candidate issues opened against older versions, runs the reproducer locally first when possible (Linux or macOS), otherwise reuses or provisions a Brev Linux box (CPU or GPU), detects behavior that was intentionally changed, scores confidence, and posts an evidence-backed comment with a label (fixed-on-latest, wontfix-by-design, or verify-inconclusive). Tag-only — never auto-closes. Brev verification is Linux-only in v1; Windows and integration-token-dependent issues are skipped. Trigger keywords - verify stale, verify fixed, reproduce on latest, stale issue, old bug, fixed-on-latest, wontfix-by-design, verify-inconclusive, drain backlog, brev verify. +description: Verify whether old NVIDIA/NemoClaw bug reports still reproduce against the latest tag. Picks candidate issues opened against older versions, runs the reproducer locally first when possible (Linux or macOS), otherwise reuses or provisions a Brev Linux box (CPU or GPU), detects behavior that was intentionally changed, scores confidence, and posts an evidence-backed comment with a label (fixed-on-latest, wontfix, or verify-inconclusive). Tag-only — never auto-closes. Brev verification is Linux-only in v1; Windows and integration-token-dependent issues are skipped. Trigger keywords - verify stale, verify fixed, reproduce on latest, stale issue, old bug, fixed-on-latest, wontfix, verify-inconclusive, drain backlog, brev verify. user_invocable: true --- @@ -71,7 +71,7 @@ Apply these rules in order. Drop any issue that fails a rule. **Idempotency:** drop if **either** of these is true: -- The issue carries a `fixed-on-latest`, `verify-inconclusive`, or `wontfix-by-design` label. (Cleared by the release sweep in `nemoclaw-maintainer-cut-release-tag` so the issue re-opens on each release.) +- The issue carries a `fixed-on-latest` or `verify-inconclusive` label. (Cleared by the release sweep in `nemoclaw-maintainer-cut-release-tag` so the issue re-opens on each release.) The by-design path uses the existing repo `wontfix` label, which is already covered by the issue-type skip rule above — no separate idempotency clause needed for that path. - A comment matching `<!-- nemoclaw-verify-stale v\d+ YYYY-MM-DD -->` was posted **within the last 7 days**. The regex matches any marker version (`v1`, `v2`, …) so future skill versions can re-verify older-marked issues by tightening the regex (e.g. require a specific marker version). The marker carries a date so the candidate filter can apply a TTL — useful for the still-reproduces case (Step 9), where no label is applied and we want next week's run to re-verify rather than skip forever. **Candidate rule:** keep the issue if **either**: @@ -514,7 +514,7 @@ Re-run every grep / git / `gh` command cited in the evidence blocks before compo - **Skip the Step 9 score table** entirely. The "exit 0 + expected output" axis doesn't apply when the expected output is no longer the contract. - **Skip Brev provisioning** if the signal fires before Step 7 — a remote run would just confirm what static analysis already proved. (Signals 2 and 3 can run as soon as the reported version is parsed in Step 4.) -- **Apply label `wontfix-by-design`** (`gh label create wontfix-by-design ...` if it doesn't exist; coordinate with maintainers on the canonical name before first use). +- **Apply label `wontfix`** (the existing repo label, not a verify-stale-specific variant). `wontfix` is already in the Step 3 issue-type skip list, so a labelled issue is automatically excluded from future runs without needing a separate idempotency clause. - **Use the by-design comment template below** instead of the standard Step 10 template. - **@-mention the reporter** so they can object if the framing is wrong. - **Never auto-close.** A maintainer pulls the trigger, same as the other label paths. @@ -523,11 +523,14 @@ Re-run every grep / git / `gh` command cited in the evidence blocks before compo Mandatory sections in this order. Omit only the sections explicitly noted as omittable. +**Tag-anchoring rule.** All `file:line` citations in the rendered comment MUST refer to the verified-on tag (e.g., `v0.0.35`), not the maintainer's working `HEAD`. Lines drift between tags and main; pinning to the tag keeps the citations reproducible by anyone reading the comment later. When greping for evidence, use `git grep -n "<symbol>" "$LATEST" -- ...` rather than working-tree grep. + ````markdown ## Stale-issue verification — behavior is by-design **Reported on:** v0.0.<X> **Verified on:** v0.0.<Y> (PR #<NNNN> first shipped in v0.0.<Z>) +**Verification mode:** static analysis at the verified-on tag — no runtime reproduction. Step 8.5 by-design short-circuits Brev provisioning because the responsible code change is already proven by the diff between `$REPORTED_VERSION` and `$LATEST`. **Outcome:** symptom reproduces against the reproducer as filed, but the implicated behavior was intentionally changed. ### What's structurally fixed @@ -827,7 +830,7 @@ After each issue (verified, inconclusive, by-design, or infra-failed), append to **Latest install:** succeeded | failed (infra error) **Latest result:** not-reproduced (clean) | still-reproduces | partial / flake | n/a (skipped 8d) **Confidence:** 88 / 100 | n/a (still-reproduces) -**Label applied:** fixed-on-latest | verify-inconclusive | wontfix-by-design | none (still-reproduces) | none (infra) +**Label applied:** fixed-on-latest | verify-inconclusive | wontfix | none (still-reproduces) | none (infra) **Brev wall time (approx):** N min --- @@ -850,7 +853,7 @@ At end of a batch session, prepend a session summary: ## YYYY-MM-DD — Verify Session **Issues considered:** N **Verified `fixed-on-latest`:** N -**Marked `wontfix-by-design`:** N +**Marked `wontfix` (by-design path):** N **Marked `verify-inconclusive`:** N **Local-first short-circuits (no Brev cost):** N **Skipped (Windows / macOS / integration / no version):** N @@ -883,4 +886,4 @@ Never stage or commit the log to the NemoClaw repo. ## Companion Behavior -`nemoclaw-maintainer-cut-release-tag` sweeps `fixed-on-latest`, `verify-inconclusive`, and `wontfix-by-design` from all open issues at release time. Without that sweep, "latest" drifts and verifications go stale silently. +`nemoclaw-maintainer-cut-release-tag` sweeps `fixed-on-latest` and `verify-inconclusive` from all open issues at release time. Without that sweep, "latest" drifts and verifications go stale silently. The by-design path uses the existing repo `wontfix` label; that label is **not** swept (it's also applied for non-skill reasons such as scope or priority decisions, and clearing it would erase human triage work). From ea6be1308cf8a6e83ef44a4fc559bb81a8a45921 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Wed, 6 May 2026 16:59:38 -0700 Subject: [PATCH 15/40] fix(verify-stale): require markdown-linked citations and link resolution check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bare `file:line` paths in comments force the reader to navigate manually — that's a usability bug, not a stylistic preference. Comments posted by the skill should be self-serving artifacts: a maintainer or QA reader should be able to click any citation and land on the exact content at the verified-on tag. Extend the tag-anchoring rule above the by-design template into a "tag-anchoring + linking rule" with concrete formats for file blobs, file:line, file:line-range, commit SHAs, and test paths. Note that bare `#NNNN` PR/issue references already auto-link in same-repo comments. Strengthen Step 8.5d into two passes: evidence (re-run cited commands) and link (resolve at least one rendered link per evidence section via `gh api .../contents/<path>?ref=<tag>` or `curl -fsI`). A 404 on a citation suggests verification that didn't actually happen — worse than no link at all. Bail to verify-inconclusive on any failure. Surfaced while reviewing the rendered #2168 comment: the citations were correct and tag-anchored but not clickable. Better caught now than after the first batch run posts a wall of bare paths. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 23 +++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 48df8291e8d..b740def61d0 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -508,7 +508,13 @@ Cite at most three concrete test paths. If none exist, omit the section — do n ### Step 8.5d: Self-verification pass before posting -Re-run every grep / git / `gh` command cited in the evidence blocks before composing the comment. If any cited `file:line`, commit SHA, or quoted output doesn't reproduce on a fresh invocation, **stop and revise** — or bail to `verify-inconclusive` if the discrepancy can't be resolved. The cost of an incorrect "I checked and X is gone" claim in a public comment is higher than spending 30 seconds re-checking. This step exists because LLMs can confidently overstate; mechanical re-verification catches it. +Two passes, both required. + +**Evidence pass.** Re-run every grep / git / `gh` command cited in the evidence blocks. If any cited `file:line`, commit SHA, or quoted output doesn't reproduce on a fresh invocation, **stop and revise** — or bail to `verify-inconclusive` if the discrepancy can't be resolved. + +**Link pass.** Resolve at least one rendered markdown link from each section that has them — `What's structurally fixed`, `Vestigial references`, `Existing CI coverage`. Use `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 content if the path exists at the tag, 404 otherwise) or `curl -fsI <blob-url>` (returns 200 if the blob renders). A broken link is worse than no link — it suggests verification work that didn't actually happen. + +The cost of an incorrect "I checked and X is gone" claim in a public comment, or a 404 on a citation, is higher than spending a minute re-checking. This step exists because LLMs can confidently overstate and confidently invent paths; mechanical re-verification catches both. ### Step 8.5e: If any signal fires @@ -523,7 +529,20 @@ Re-run every grep / git / `gh` command cited in the evidence blocks before compo Mandatory sections in this order. Omit only the sections explicitly noted as omittable. -**Tag-anchoring rule.** All `file:line` citations in the rendered comment MUST refer to the verified-on tag (e.g., `v0.0.35`), not the maintainer's working `HEAD`. Lines drift between tags and main; pinning to the tag keeps the citations reproducible by anyone reading the comment later. When greping for evidence, use `git grep -n "<symbol>" "$LATEST" -- ...` rather than working-tree grep. +**Tag-anchoring + linking rule.** Every `file:line` citation, commit SHA, and test-path reference in the rendered comment MUST be a clickable markdown link to the verified-on tag (e.g., `v0.0.35`), not the maintainer's working `HEAD`. Lines drift between tags and main; tag-anchored links keep the citations reproducible by anyone reading the comment months later. Bare paths force the reader to navigate manually — that's a usability bug, not a stylistic preference. + +Use these exact link formats: + +- File only: `[src/lib/onboard.ts](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/onboard.ts)` +- File:line: `[src/lib/onboard.ts:4965](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/onboard.ts#L4965)` +- File:line-range: `[src/lib/commands/sandbox/connect.ts:25-31](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/commands/sandbox/connect.ts#L25-L31)` +- Commit SHA: `[5956a61](https://github.com/NVIDIA/NemoClaw/commit/5956a612e18047b9ab85b3a7e89f6b5dedb29190)` — short SHA as the link text, full SHA in the URL +- Test file: `[test/e2e/test-double-onboard.sh](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/test/e2e/test-double-onboard.sh)` +- PR/issue references: bare `#NNNN` works — GitHub auto-links these in comments on the same repo, no manual URL needed. + +When greping for evidence, use `git grep -n "<symbol>" "$LATEST" -- ...` so the line numbers match the tagged blob. Then construct each link from `<file path> + verified-on tag + line number`. + +The Step 8.5d self-verification pass MUST resolve at least one rendered link (e.g., `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=v0.0.35` or a `curl -fsI` to the blob URL) and confirm it returns the expected file. A broken link defeats the purpose of including the citation. If any link fails to resolve, fix it or bail to `verify-inconclusive`. ````markdown ## Stale-issue verification — behavior is by-design From 1a20971aa3d6d72e96b87239eb6b23e6b151746b Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Wed, 6 May 2026 17:06:54 -0700 Subject: [PATCH 16/40] fix(verify-stale): align skill label vocabulary with actual repo labels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Discovered while applying the by-design label to issue #2168: the skill referenced `wontfix` and `needs-info` as bare strings, but the repo's canonical labels are `status: wont-fix` and `status: needs-info` (with prefix and hyphen). Same shape in Step 3's issue-type skip list — issues labelled `status: wont-fix` or `status: needs-info` were NOT being filtered out at candidate-time as the spec intended, because the skip list keys didn't match the live labels. Three coordinated fixes: 1. Step 3 issue-type skip list now uses `status: wont-fix` and `status: needs-info`. Annotate the rule so future readers know not to substitute the bare forms. 2. Step 8.5 by-design action applies `status: wont-fix` (with quoting on the CLI). Step 12 log entry, session summary, frontmatter description, and Companion Behavior section in both the verify-stale skill and `nemoclaw-maintainer-cut-release-tag` updated to match. 3. Step 6.5 preconditions now verifies all expected labels exist on the repo with `gh label list` before any later step tries to apply them. This is the gap that bit us on #2168 — the comment posted but the label couldn't be applied because the spec name didn't match. Failing fast in 6.5 means the run aborts before any Brev cost or comment posting can happen against a misconfigured label vocabulary. Bare `wontfix` / `won't fix` / etc. are still in Signal 1's detection regex — that's intentional, those are SEARCH PATTERNS for what maintainers might write in free-text comments, not label names. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../SKILL.md | 2 +- .../nemoclaw-maintainer-verify-stale/SKILL.md | 27 ++++++++++++++----- 2 files changed, 21 insertions(+), 8 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md index 118eec08dbc..5091b5cc4cb 100644 --- a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md @@ -129,7 +129,7 @@ Confirm both tags point to the same commit on the remote. ## Step 7: Sweep Stale-Issue Verification Labels -Strip `fixed-on-latest` and `verify-inconclusive` from all open issues so the next `nemoclaw-maintainer-verify-stale` run re-evaluates against the new release. Without this sweep, "latest" drifts and verifications go silently stale. The skill's by-design path uses the existing repo `wontfix` label, which is **not** swept here — `wontfix` is also applied for non-skill reasons (scope, priority, dup), so clearing it would erase human triage work. +Strip `fixed-on-latest` and `verify-inconclusive` from all open issues so the next `nemoclaw-maintainer-verify-stale` run re-evaluates against the new release. Without this sweep, "latest" drifts and verifications go silently stale. The skill's by-design path uses the existing repo `status: wont-fix` label, which is **not** swept here — that label is also applied for non-skill reasons (scope, priority, dup decisions), so clearing it would erase human triage work. ```bash for label in fixed-on-latest verify-inconclusive; do diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index b740def61d0..83096ebd9f3 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -1,6 +1,6 @@ --- name: nemoclaw-maintainer-verify-stale -description: Verify whether old NVIDIA/NemoClaw bug reports still reproduce against the latest tag. Picks candidate issues opened against older versions, runs the reproducer locally first when possible (Linux or macOS), otherwise reuses or provisions a Brev Linux box (CPU or GPU), detects behavior that was intentionally changed, scores confidence, and posts an evidence-backed comment with a label (fixed-on-latest, wontfix, or verify-inconclusive). Tag-only — never auto-closes. Brev verification is Linux-only in v1; Windows and integration-token-dependent issues are skipped. Trigger keywords - verify stale, verify fixed, reproduce on latest, stale issue, old bug, fixed-on-latest, wontfix, verify-inconclusive, drain backlog, brev verify. +description: Verify whether old NVIDIA/NemoClaw bug reports still reproduce against the latest tag. Picks candidate issues opened against older versions, runs the reproducer locally first when possible (Linux or macOS), otherwise reuses or provisions a Brev Linux box (CPU or GPU), detects behavior that was intentionally changed, scores confidence, and posts an evidence-backed comment with a label (fixed-on-latest, status wont-fix, or verify-inconclusive). Tag-only — never auto-closes. Brev verification is Linux-only in v1; Windows and integration-token-dependent issues are skipped. Trigger keywords - verify stale, verify fixed, reproduce on latest, stale issue, old bug, fixed-on-latest, status wont-fix, verify-inconclusive, drain backlog, brev verify. user_invocable: true --- @@ -61,7 +61,7 @@ This is the version the skill will verify against. Record it — every comment m Apply these rules in order. Drop any issue that fails a rule. **Issue-type allowlist:** must have `bug` label. -**Issue-type skip:** drop if any of `enhancement`, `documentation`, `wontfix`, `needs-info`, `security`. +**Issue-type skip:** drop if any of `enhancement`, `documentation`, `status: wont-fix`, `status: needs-info`, `security`. Use the canonical repo label names — bare `wontfix` / `needs-info` are NOT the repo's labels (verified via `gh label list`); the actual labels carry a `status:` prefix and a hyphen. **Platform skip (Linux-only in v1):** drop if any of `Platform: Windows/WSL`, `Platform: MacOS`, `Platform: macOS`. Keep `Platform: Ubuntu`, `Platform: DGX Spark`, `Platform: GB10`, `Platform: All`, or no platform label. @@ -71,7 +71,7 @@ Apply these rules in order. Drop any issue that fails a rule. **Idempotency:** drop if **either** of these is true: -- The issue carries a `fixed-on-latest` or `verify-inconclusive` label. (Cleared by the release sweep in `nemoclaw-maintainer-cut-release-tag` so the issue re-opens on each release.) The by-design path uses the existing repo `wontfix` label, which is already covered by the issue-type skip rule above — no separate idempotency clause needed for that path. +- The issue carries a `fixed-on-latest` or `verify-inconclusive` label. (Cleared by the release sweep in `nemoclaw-maintainer-cut-release-tag` so the issue re-opens on each release.) The by-design path uses the existing repo `status: wont-fix` label, which is already covered by the issue-type skip rule above — no separate idempotency clause needed for that path. - A comment matching `<!-- nemoclaw-verify-stale v\d+ YYYY-MM-DD -->` was posted **within the last 7 days**. The regex matches any marker version (`v1`, `v2`, …) so future skill versions can re-verify older-marked issues by tightening the regex (e.g. require a specific marker version). The marker carries a date so the candidate filter can apply a TTL — useful for the still-reproduces case (Step 9), where no label is applied and we want next week's run to re-verify rather than skip forever. **Candidate rule:** keep the issue if **either**: @@ -204,6 +204,19 @@ brev ls --json >/dev/null 2>&1 || { exit 1 } +# Repo labels exist — Step 8.5 / Step 10 can't apply a label that doesn't exist. Check +# canonical label names against the live repo so a mismatch fails fast (issue #2168 hit this: +# spec called the label `wontfix`, but the actual repo label is `status: wont-fix`). +EXPECTED_LABELS=("fixed-on-latest" "verify-inconclusive" "status: wont-fix") +LIVE_LABELS=$(gh label list --repo NVIDIA/NemoClaw --limit 200 --json name --jq '.[].name') +for label in "${EXPECTED_LABELS[@]}"; do + printf '%s\n' "$LIVE_LABELS" | grep -Fxq "$label" || { + echo "ERROR: expected label not on repo: '$label'" + echo " create it with: gh label create '$label' --repo NVIDIA/NemoClaw" + exit 1 + } +done + # Install URL reachable — fails fast instead of mid-Brev-run if the host is down or the URL changed. INSTALL_URL=${NEMOCLAW_INSTALL_URL:-https://nemoclaw.nvidia.com/install.sh} curl -fsI "$INSTALL_URL" >/dev/null 2>&1 || { @@ -520,7 +533,7 @@ The cost of an incorrect "I checked and X is gone" claim in a public comment, or - **Skip the Step 9 score table** entirely. The "exit 0 + expected output" axis doesn't apply when the expected output is no longer the contract. - **Skip Brev provisioning** if the signal fires before Step 7 — a remote run would just confirm what static analysis already proved. (Signals 2 and 3 can run as soon as the reported version is parsed in Step 4.) -- **Apply label `wontfix`** (the existing repo label, not a verify-stale-specific variant). `wontfix` is already in the Step 3 issue-type skip list, so a labelled issue is automatically excluded from future runs without needing a separate idempotency clause. +- **Apply label `status: wont-fix`** (the existing repo label — quote it on the CLI: `gh issue edit <num> --add-label "status: wont-fix"`). It's already in the Step 3 issue-type skip list, so a labelled issue is automatically excluded from future runs without needing a separate idempotency clause. - **Use the by-design comment template below** instead of the standard Step 10 template. - **@-mention the reporter** so they can object if the framing is wrong. - **Never auto-close.** A maintainer pulls the trigger, same as the other label paths. @@ -849,7 +862,7 @@ After each issue (verified, inconclusive, by-design, or infra-failed), append to **Latest install:** succeeded | failed (infra error) **Latest result:** not-reproduced (clean) | still-reproduces | partial / flake | n/a (skipped 8d) **Confidence:** 88 / 100 | n/a (still-reproduces) -**Label applied:** fixed-on-latest | verify-inconclusive | wontfix | none (still-reproduces) | none (infra) +**Label applied:** fixed-on-latest | verify-inconclusive | status: wont-fix | none (still-reproduces) | none (infra) **Brev wall time (approx):** N min --- @@ -872,7 +885,7 @@ At end of a batch session, prepend a session summary: ## YYYY-MM-DD — Verify Session **Issues considered:** N **Verified `fixed-on-latest`:** N -**Marked `wontfix` (by-design path):** N +**Marked `status: wont-fix` (by-design path):** N **Marked `verify-inconclusive`:** N **Local-first short-circuits (no Brev cost):** N **Skipped (Windows / macOS / integration / no version):** N @@ -905,4 +918,4 @@ Never stage or commit the log to the NemoClaw repo. ## Companion Behavior -`nemoclaw-maintainer-cut-release-tag` sweeps `fixed-on-latest` and `verify-inconclusive` from all open issues at release time. Without that sweep, "latest" drifts and verifications go stale silently. The by-design path uses the existing repo `wontfix` label; that label is **not** swept (it's also applied for non-skill reasons such as scope or priority decisions, and clearing it would erase human triage work). +`nemoclaw-maintainer-cut-release-tag` sweeps `fixed-on-latest` and `verify-inconclusive` from all open issues at release time. Without that sweep, "latest" drifts and verifications go stale silently. The by-design path uses the existing repo `status: wont-fix` label; that label is **not** swept (it's also applied for non-skill reasons such as scope or priority decisions, and clearing it would erase human triage work). From ef1d9809efc28053e92742a01b9dbbf1dcfc522e Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Wed, 6 May 2026 17:16:42 -0700 Subject: [PATCH 17/40] fix(verify-stale): default cap to 60 min and bootstrap reproducer deps for faithfulness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes that landed before the first real Brev E2E run (#2007). The previous wallclock cap split — 25 min default with a 60 min extension keyed off "memory leak" / "over time" / "eventually" keywords — optimised for the wrong constraint. Most issues fit comfortably under 60 min once you account for two full install passes plus comprehensive resets, and the keyword-based extension forced re-runs whenever a real install or bootstrap took longer than the optimistic 25 min budget. Bump the cap to a single 60 min default and drop the keyword extension; bugs that genuinely require more than an hour fall out of v1 scope. Add a new Step 8a.5 for bootstrapping reproducer dependencies that the stock Brev image doesn't ship — local model servers (Ollama, vLLM), provider runtimes, third-party CLIs. Default policy is maximum faithfulness: install the actual dependency the reporter used rather than substituting a stub. Substituting trades faithfulness for speed, and on a 60 min budget that trade is rarely worth it; it almost always introduces a confound that makes the verdict less trustworthy. Document the canonical Ollama and vLLM bootstraps with specific attention to daemon survival between `brev exec` calls (Ollama's installer registers a systemd service; vLLM uses nohup + log file). Bootstrap runs once before Step 8b's baseline and is reused for Step 8d's latest run — model downloads are expensive and external state, so the comprehensive reset must explicitly leave them alone. Carve out one substitution case: providers that require API keys the skill cannot safely supply (NIM, OpenAI, Anthropic). Stubbing a key defeats faithfulness anyway, and a real key shouldn't sit in a verify-stale run. Substitution applies the same -30 LLM-synth penalty as Step 8b and must be documented in the rendered comment. Bootstrap failure escalates to Step 11 infra failure — do NOT silently substitute, since the maintainer opted into faithfulness for a reason. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 44 ++++++++++++++++++- 1 file changed, 42 insertions(+), 2 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 83096ebd9f3..04e3a70675b 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -321,9 +321,9 @@ echo ">>> Brev instance: $INSTANCE_NAME (provisioned_new=$PROVISIONED_NEW; manua trap '[ "$PROVISIONED_NEW" = "1" ] && brev delete "$INSTANCE_NAME" >/dev/null 2>&1 || true' EXIT ``` -Wallclock cap per verification: **25 minutes** to accommodate two installs (reported version baseline + latest). If a provisioned box isn't ready in time, abort and treat as an infra failure (Step 11). +Wallclock cap per verification: **60 minutes** default. The cap accommodates two full install passes (baseline + latest), comprehensive resets between them, and any reproducer dependency bootstrapping (Step 8a.5) — most of which run sequentially against a single Brev box. Bugs that genuinely require more than an hour to manifest fall out of v1 scope; if a provisioned box isn't ready in time, abort and treat as an infra failure (Step 11). -**Extended budget for time-sensitive bugs.** If the issue body contains keywords suggesting the bug only manifests over time (`after N minutes`, `after N requests`, `eventually`, `over time`, `memory leak`, `long-running`, `idle for`), bump the cap to **60 minutes**. Detection is simple keyword match. Hard ceiling at 60 min — bugs that genuinely require hours fall out of v1 scope. +The previous design had a 25-min default with a 60-min extension for time-sensitive bugs (`memory leak`, `over time`, etc.). That split optimised for the wrong constraint — most issues fit comfortably under 60 min, and the keyword-based extension forced re-runs whenever a real install or bootstrap took longer than the optimistic 25-min budget. Single 60-min cap removes that paper cut. --- @@ -379,6 +379,46 @@ brev exec "$INSTANCE_NAME" "nemoclaw --version" If install fails (old releases rot — installer URLs, deps, OS images all drift over time), set `BASELINE_INSTALL_FAILED=1` and **skip 8b/8c**, going straight to 8d. Note "baseline-install-skipped" in the final comment. Step 9's scoring rule handles the degraded mode. +### Step 8a.5: Bootstrap reproducer dependencies + +Brev's stock CPU images ship with NemoClaw installable but not the broader ecosystem the reproducer may need — local model servers (Ollama, vLLM), inference providers, third-party CLIs. **Default to maximum faithfulness: install the actual dependency the reporter used rather than substituting a stub.** Substituting trades faithfulness for speed; that trade is rarely worth it on a 60-min budget, and it almost always introduces a confound that makes the verdict less trustworthy. + +**When to bootstrap (not substitute):** + +- The reproducer references a specific model/server runtime (`NEMOCLAW_PROVIDER=ollama`, `NEMOCLAW_PROVIDER=vllm`, etc.). +- The reproducer references a specific model name with a tag (`nemotron-3-nano:4b`, `llama3:8b`, etc.). +- The reporter's environment in the issue body shows a configured provider (e.g., `OpenShell CLI: 0.0.26` plus an Ollama running on host). + +**When to substitute (with -30 penalty):** + +- Provider requires an API key the skill cannot safely supply (NIM, OpenAI, Anthropic, etc.). Stubbing a key won't pass validation faithfully and a real key shouldn't sit in a verify-stale run. Apply the -30 penalty (treat as synth-repro per Step 8b) and document the substitution in the comment. +- The bug is *provably* independent of the dependency (e.g., a CLI argument-parsing bug that errors before any provider runs). Note this explicitly in the comment. + +**Canonical bootstraps:** + +```bash +# Ollama + a specific model. +# The Ollama installer registers a systemd service (`ollama.service`) so the +# daemon survives between brev exec calls. +brev exec "$INSTANCE_NAME" "curl -fsSL https://ollama.com/install.sh | sh" +brev exec "$INSTANCE_NAME" "sudo systemctl start ollama && sleep 3" +brev exec "$INSTANCE_NAME" "ollama pull <model>" +brev exec "$INSTANCE_NAME" "ollama list" # confirm before continuing +``` + +```bash +# vLLM + a model (HuggingFace-hosted). +brev exec "$INSTANCE_NAME" "pip install --quiet vllm" +brev exec "$INSTANCE_NAME" "nohup python -m vllm.entrypoints.openai.api_server --model <model> --host 127.0.0.1 --port 8000 >/var/log/vllm.log 2>&1 &" +brev exec "$INSTANCE_NAME" "sleep 30 && curl -fsS http://127.0.0.1:8000/v1/models" +``` + +Bootstrap **once before Step 8b's baseline run** and reuse for Step 8d's latest run. Don't reset Ollama/vLLM state between baseline and latest in the comprehensive reset — model downloads are expensive and unrelated to the NemoClaw install. Adjust the reset script to skip these external services explicitly if needed. + +**If bootstrap fails** (network issue pulling the model, service won't start, etc.), this is an infra failure — abort to Step 11. Do not silently substitute; the user opted into faithfulness for a reason. + +--- + ### Step 8b: Run reproducer on baseline, compare to issue symptom If `./reproducer.sh` exists (verbatim from Step 6), run it. Otherwise synth on demand from the issue body (apply −30 penalty now, locked in for the rest of the run). From d0fcf5ece334b1bd135e1a8d8aabc7e06366fec3 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 11:43:20 -0700 Subject: [PATCH 18/40] fix(verify-stale): mandate reporter @-mention and 400 to 500-word target on every comment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two requirements surfaced from the #2007 e2e run that should be enforced by the skill, not by per-session memory or by the prompter remembering. Mandatory reporter @-mention with confirmation language. The skill cannot independently confirm a closed-as-fixed verdict — only the reporter knows whether their original symptom is gone in their environment. The @-mention is what converts a "skill says it is fixed" claim into actionable confirmation work for QA. Add the explicit closing block (canonical wording: "please confirm the symptom is gone on a recent build and reopen with a fresh reproducer if you observe otherwise") to all three Step 10 templates: fixed-on-latest, still-reproduces, and the Step 8.5 by-design template. The previous "If this verification is wrong, please reopen..." line was passive; the new line names the reporter and asks them to act. Length target. Default rendered comments to 400-500 words. The evidence table or by-design fixed/vestigial sections are the hero; everything else has to either change the reader's mind about the verdict or be deleted. The first #2007 draft ran ~750 words and the user pushed back explicitly; encoding the cap into Step 10 means future agents do not start from scratch on length each run. Surfaced from: e2e Brev verification run on issue #2007 (the first real Brev-track exercise of the skill end to end). Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 04e3a70675b..6fdef6d8a72 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -630,7 +630,7 @@ The new workflow is `<one-sentence: how to do what the user was trying to do>`. ### Recommendation -@<reporter> — recommend closing as "won't fix / by design". If a related symptom (e.g. `<related failure mode from above>`) is hitting you on ≥ v0.0.<Z>, please file a fresh issue with a v0.0.<Z>+ reproducer. +@<reporter> — please confirm the by-design framing is correct (the implicated `<symbol>` was intentionally removed, the original reproducer can no longer execute) and close as "won't fix / by design" if you agree. If a related symptom (e.g. `<related failure mode from above>`) is hitting you on ≥ v0.0.<Z>, please file a fresh issue with a v0.0.<Z>+ reproducer. `<NVBugs cross-ref line — see below>` @@ -764,6 +764,14 @@ Transcripts and synth-repro scripts are already plain text and skip the pre-pass **File paths under the reporter's home directory** (`/Users/<name>/`, `/home/<name>/`) → replace with `~/`. Run last; catches incidental username PII. +**Length target.** Default rendered comment to **400–500 words**. The evidence table (or by-design "What's structurally fixed" + "Vestigial references" sections) is the hero. Strip architectural prose "for QA reference," PR-attribution caveats beyond one sentence, and closing reopen-instructions boilerplate. If a comment runs past 500 words, cut everything that doesn't directly support the verdict — every section needs to either change a reader's mind about the verdict or be deleted. + +**Mandatory closing block — reporter @-mention with confirmation language.** Every template below ends with an explicit @-mention of the original reporter using this exact shape: + +> @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. + +The skill cannot independently confirm a closed-as-fixed verdict — only the reporter knows whether their original symptom is gone in their environment. The @-mention is what converts a "skill says it's fixed" claim into actionable confirmation work for QA. Customize `<Z>` per case (the version that shipped the fix or `$LATEST`), but never omit the line. + **Comment template (fixed / inconclusive — bug not reproduced on latest):** ````markdown @@ -811,7 +819,7 @@ Transcripts and synth-repro scripts are already plain text and skip the pre-pass </details> -If this verification is wrong, please reopen the issue with a comment and the skill will re-verify on the next release. +@<reporter> — please confirm the symptom is gone on a recent build (≥ v0.0.<Z>) and reopen with a fresh reproducer if you observe otherwise. <!-- nemoclaw-verify-stale v1 2026-05-12 --> ```` @@ -829,6 +837,8 @@ The skill ran the reported reproducer on v0.0.34 and observed the same bug sympt No label applied. Will re-verify automatically next weekly run; if a fix lands in the interim, the next pass catches it. +@<reporter> — please confirm the symptom still matches your observation on v0.0.<Y> and reopen with any updated reproducer or environment details if it has shifted. + <details><summary>Baseline transcript (validated reproducer)</summary> ```text From 296f7cd4f82a3d824e616ad9444103c3b5e0ea5b Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 11:51:39 -0700 Subject: [PATCH 19/40] fix(verify-stale): land six post-#2007 hardening fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six concrete gaps that surfaced during the first end-to-end Brev run (issue #2007). Each almost produced a wrong verdict or a wasted run; each is now encoded so the next agent doesn't have to re-discover. 1. Step 9 baseline-validation cap-removal rule was too permissive. The `unless commits-touched OR PR-mention also fires` escape hatch let inferred fix evidence override the absence of a baseline run, which produced a misleading 100/100 on #2007 despite zero baseline confirmation. Tightened: cap at 84 holds regardless of corroboration; PR-search signals raise the score within the cap, never past it. Step 10 now requires an explicit one-line caveat naming the cap and the reason in the rendered Verdict section. 2. Step 6.5 install URL default switched from `nemoclaw.nvidia.com` (NVIDIA-internal, doesn't resolve from Brev) to `www.nvidia.com/ nemoclaw.sh` (public Akamai 301 redirect). Brev runs were dying at bootstrap on the wrong host. 3. Step 7 CPU SKU picker now biases by reproducer-implied memory needs. On #2007 the cheapest 2 GB SKU couldn't load a 4.8 GiB Ollama probe; onboard failed at provider validation and we burned ~25 min before re-provisioning a 16 GB box. Adds a `CPU_RAM_FLOOR` env var and a memory-floor heuristic (16 GB when reproducer references a model server, 8 GB for sandbox onboarding without a model, 4 GB for pure-CLI bugs). 4. Step 11 failure taxonomy now has a "baseline-build rot" bucket distinct from "binary install rot." Same `BASELINE_INSTALL_FAILED=1` flag and same downstream cap-and-degrade behavior, but failures at the in-image Dockerfile build phase (what we hit on v0.0.18 — the `.openclaw-data/workspace/media` symlink layer, removed entirely by #2227) get a separate label so reviewers can see *why* the old image no longer builds without re-running. 5. New Step 8a.5b documents the two non-obvious `brev exec` quirks that reproducer scripts have to handle every time: PATH does not include `~/.local/bin` in non-login shells (so reproducers must `export PATH` at the top, or callers must wrap with `bash -lc`); and the docker group requires `sg docker -c '...'` because adding the user via `usermod -aG` doesn't take effect within the same Brev session. 6. New Step 8d.5 architectural-drift check. When the diff between `$REPORTED_VERSION` and `$LATEST` touches the *tool* the reproducer's expected output depends on (e.g. `openshell forward` between v0.0.18 and v0.0.35), don't trust the reproducer's surface alone — multi-axis verification on OS-level surfaces (host listeners, NAT rules, docker ports, SSH tunnels, etc.) is required before claiming fixed-on-latest. This is the five-axis pattern we used to confirm #2007 wasn't a false positive; codifies it as a check the skill applies whenever pickaxe shows the reproducer's tool was reworked. Surfaced from: end-to-end Brev verification of issue #2007. Build the skill, exercise it on a real issue, fix what breaks, repeat — every fix here came from a concrete failure mode in one real run. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 107 ++++++++++++++++-- 1 file changed, 99 insertions(+), 8 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 6fdef6d8a72..7540e529742 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -218,7 +218,10 @@ for label in "${EXPECTED_LABELS[@]}"; do done # Install URL reachable — fails fast instead of mid-Brev-run if the host is down or the URL changed. -INSTALL_URL=${NEMOCLAW_INSTALL_URL:-https://nemoclaw.nvidia.com/install.sh} +# The default is the public Akamai-hosted entry (301-redirects to the actual installer). The +# `nemoclaw.nvidia.com` host that earlier drafts pointed to is NVIDIA-internal and does not +# resolve from Brev; surfaced during the #2007 e2e run. +INSTALL_URL=${NEMOCLAW_INSTALL_URL:-https://www.nvidia.com/nemoclaw.sh} curl -fsI "$INSTALL_URL" >/dev/null 2>&1 || { echo "ERROR: install URL not reachable: $INSTALL_URL" echo "Set NEMOCLAW_INSTALL_URL or check https://nemoclaw.nvidia.com is up." @@ -303,11 +306,23 @@ else # (>=20GB VRAM, >=500GB disk, compute >=8.0). Override with --type if needed. brev create "$INSTANCE_NAME" else - # CPU case: pick the cheapest stoppable Linux SKU at runtime so the skill - # doesn't rot when SKUs change. Override by exporting VERIFY_STALE_CPU_TYPE. + # CPU case: pick the cheapest stoppable Linux SKU at runtime so the skill doesn't rot when + # SKUs change. Bias the floor by reproducer-implied memory needs — the cheapest 2 GB SKU + # cannot load a 4.8 GiB Ollama probe, and onboard fails at provider validation before any + # sandbox-creation code runs. Surfaced during the #2007 e2e run (wasted ~25 min on a 2 GB + # box that couldn't load `nemotron-3-nano:4b`). + # + # Memory floor heuristic: + # - Reproducer references Ollama or vLLM or names a model tag (e.g. `nemotron-3-nano:4b`, + # `llama3:8b`) -> floor 16 GB (covers ~5 GB model + sandbox + gateway overhead). + # - Reproducer touches sandbox onboarding without a local model server -> floor 8 GB. + # - Pure CLI-surface bug (no sandbox, no model) -> floor 4 GB. + # Override the auto-pick by exporting VERIFY_STALE_CPU_TYPE if the team has hard preferences. + CPU_RAM_FLOOR=${CPU_RAM_FLOOR:-8} CPU_TYPE=${VERIFY_STALE_CPU_TYPE:-$(brev search cpu --sort price --json \ - | jq -r '[.[] | select(.stoppable == true)] | .[0].type')} - [ -n "$CPU_TYPE" ] || { echo "ERROR: no stoppable CPU SKU available"; exit 1; } + | jq -r --argjson floor "$CPU_RAM_FLOOR" \ + '[.[] | select(.stoppable == true and .ram_gb >= $floor)] | .[0].type')} + [ -n "$CPU_TYPE" ] || { echo "ERROR: no stoppable CPU SKU with >= ${CPU_RAM_FLOOR} GB RAM"; exit 1; } brev create "$INSTANCE_NAME" --type "$CPU_TYPE" fi @@ -417,6 +432,29 @@ Bootstrap **once before Step 8b's baseline run** and reuse for Step 8d's latest **If bootstrap fails** (network issue pulling the model, service won't start, etc.), this is an infra failure — abort to Step 11. Do not silently substitute; the user opted into faithfulness for a reason. +### Step 8a.5b: Brev exec environment quirks + +Two non-obvious gotchas surfaced during the #2007 e2e run that every subsequent `brev exec` call has to handle. Encode them once here so reproducer scripts don't have to relearn each time. + +**PATH does not include `~/.local/bin` in non-login shells.** `nemoclaw`'s installer drops a shim at `~/.local/bin/nemoclaw` and updates PATH via `~/.bashrc` / `~/.profile`. `brev exec` spawns non-login, non-interactive shells that don't source those files, so a bare `brev exec "$INSTANCE" "nemoclaw --version"` returns `command not found` on a freshly-installed box. Fix: every reproducer script must explicitly export PATH at the top, OR every `brev exec` call must wrap with `bash -lc '...'`. + +```bash +# Reproducer scripts: prepend this line. +export PATH="$HOME/.local/bin:$PATH" + +# Or equivalently when calling brev exec ad-hoc: +brev exec "$INSTANCE" "bash -lc 'nemoclaw --version'" +``` + +**Docker group requires `sg docker -c '...'` after `usermod -aG`.** Adding the user to the `docker` group (`sudo usermod -aG docker ubuntu`) takes effect for new login sessions, but `brev exec` calls in the same Brev session keep the old gid. The reproducer's `nemoclaw onboard` will fail with `permission denied while connecting to /var/run/docker.sock` unless the call runs in a subshell with the docker group active. + +```bash +# Reproducer execution: wrap with sg docker. +brev exec "$INSTANCE" "sg docker -c 'bash ~/reproducer.sh'" +``` + +Both patterns appear in the canonical setup script committed alongside the skill (or are encoded in your reproducer wrapper). Don't rely on the user discovering them mid-run. + --- ### Step 8b: Run reproducer on baseline, compare to issue symptom @@ -486,6 +524,48 @@ brev shell "$INSTANCE_NAME" --- +## Step 8d.5: Architectural-Drift Check + +Cross-version verification compares two moving targets: the reproducer assumes `$REPORTED_VERSION`'s tooling surface, and `$LATEST` may have rewritten the surface entirely. If the *tool* the reproducer relies on (CLI subcommand, output table, log file location) was reworked between the two tags, an "empty / clean output on latest" can mean either "bug fixed" OR "we're looking at a deprecated tracking surface." Without this check, the latter silently registers as the former — a class of false positive. + +**Detection** — pickaxe the diff between tags for the reproducer's tool name and watch for the CLI itself being touched, not just its consumers: + +```bash +# Extract the primary verification command from the reproducer (e.g. "openshell forward list"). +TOOL=$(grep -oE '\b(openshell|nemoclaw)[[:space:]]+[a-z-]+' reproducer.sh | sort -u) + +# Pickaxe each tool name across the version range. +for t in $TOOL; do + echo "=== drift check: $t ===" + git log "$REPORTED_VERSION".."$LATEST" -S"$t" --oneline -- src/ bin/ nemoclaw/src/ 2>&1 | head -5 +done +``` + +If a tool is touched, drift is suspected. + +**Multi-axis verification** — when drift is suspected, do not rely on the reproducer's expected output alone. Pick OS-level surfaces that would show the buggy state regardless of which CLI tracks it. For port-forwarding bugs (the #2007 case), the canonical five-axis pattern: + +| # | Surface | Command | +|---|---|---| +| 1 | Reproducer's stated check | as written in the issue body | +| 2 | Host TCP listeners | `sudo ss -tlnp` | +| 3 | iptables NAT redirects | `sudo iptables -t nat -L -n` | +| 4 | Docker port mappings | `docker ps --format '{{.Names}} {{.Ports}}'` | +| 5 | Active SSH tunnels | `ps -ef \| grep 'ssh.*-L'` | + +Adapt the axes to the bug class. For filesystem bugs: `find`, `lsattr`, `stat`. For network policy bugs: `iptables -L`, container netns, gateway logs. The principle is the same — pick at least three independent surfaces that would each independently show the buggy state if it were present. + +**Action when drift is suspected:** + +- Run the multi-axis pattern after Step 8d's reproducer. +- The verdict requires **every relevant axis to be clean** — not just the reproducer's surface — before claiming `fixed-on-latest`. +- Quote the multi-axis evidence in the Step 10 comment as a table; this is exactly what makes "fixed" defensible when the original tooling no longer reflects the underlying behavior. +- If any axis still shows the buggy state, the bug is NOT fixed even if the reproducer's surface is clean. Escalate to "still reproduces" (Step 9 special case). + +**When drift is NOT suspected** (the reproducer's tool is unchanged in the version range): the reproducer's expected output is sufficient, no multi-axis verification needed. + +--- + ## Step 8.5: Detect "Behavior Changed by Design" Before scoring, check whether the symptom is intentional. Some bugs are filed against behavior that was **deliberately changed or removed** in a merged PR — running the standard rubric on these produces misleading verdicts. The symptom "still reproduces" but the right answer is "won't fix, see PR #X." Issue #2791 is the prototype: `config set` was removed in PR #2227, the reporter tested a version that already had it gone, and a standard rubric run would have buried that context under a low-confidence `verify-inconclusive` label. @@ -703,7 +783,7 @@ Apply +25 if either query returns at least one PR with `mergedAt` strictly after If neither query returns anything, **skip the +25 signal**. -**Baseline-validation gating.** The +50 weight assumes the reproducer was *validated* — i.e., it produced the bug symptom on baseline (Step 8b/8c match). If `BASELINE_INSTALL_FAILED=1` (Step 8a fall-through, baseline pass skipped), the +50 still applies but **cap the total at 84** unless commits-touched-area or merged-PR-mention also fires. Without baseline AND without corroborating evidence, the cleanest landing is the 60–84 band where the reporter is asked to confirm — we don't have enough on our own to claim ≥85. +**Baseline-validation gating.** The +50 weight assumes the reproducer was *validated* — i.e., it produced the bug symptom on baseline (Step 8b/8c match). If `BASELINE_INSTALL_FAILED=1` (Step 8a fall-through, baseline pass skipped — including the sandbox-build-rot case from Step 11), the +50 still applies but **cap the total at 84**. Corroboration signals (commits-touched-area, PR-mention) still raise the score within the cap but cannot lift it above 84. Without runtime baseline confirmation we don't have enough on our own to claim ≥85 — the cap forces the verdict into the 60–84 band where the reporter is asked to confirm. The previous draft of this rule had an "unless commits-touched OR PR-mention also fires" escape hatch that let inferred fix evidence bypass the cap entirely; that produced a misleading 100/100 on the #2007 e2e run despite zero baseline confirmation, and was tightened here. **Action (when latest run was clean — bug not reproduced):** @@ -766,6 +846,8 @@ Transcripts and synth-repro scripts are already plain text and skip the pre-pass **Length target.** Default rendered comment to **400–500 words**. The evidence table (or by-design "What's structurally fixed" + "Vestigial references" sections) is the hero. Strip architectural prose "for QA reference," PR-attribution caveats beyond one sentence, and closing reopen-instructions boilerplate. If a comment runs past 500 words, cut everything that doesn't directly support the verdict — every section needs to either change a reader's mind about the verdict or be deleted. +**Mandatory cap caveat.** When the score is capped (Step 9 baseline-validation gating, or any Step 11 degraded-mode path), the rendered Verdict section must include a one-line caveat naming the cap and the reason. Example: `Capped at 84 because Step 9's baseline-validation gate did not run (sandbox-build rot on v0.0.18: Dockerfile symlink layer removed by #2227).` Don't make readers reverse-engineer why the score didn't go higher — name it. + **Mandatory closing block — reporter @-mention with confirmation language.** Every template below ends with an explicit @-mention of the original reporter using this exact shape: > @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. @@ -887,10 +969,19 @@ The next weekly run retries naturally. **Baseline-install failure** (Step 8a, reported version won't install on a modern image): not a hard failure — degraded mode. - Set `BASELINE_INSTALL_FAILED=1`, skip 8b/8c, jump to 8d. -- Step 9 applies the score cap (max 84) unless corroborating evidence fires. +- Step 9 applies the score cap (max 84) — corroboration signals raise the score within the cap but cannot lift past it. - Note "baseline-install-skipped" in the final comment so a reviewer knows the verification ran without the script-validation gate. -This degradation is expected — old releases rot. We still want to extract whatever signal we can from the latest run plus PR/commit evidence, just at a more conservative confidence ceiling. +**Baseline-build failure** (Step 8a binary install succeeded, but the in-image `Dockerfile` build during sandbox creation failed on a layer that was structurally removed in a later release): also degraded mode, distinct from binary install rot. Surfaced during the #2007 e2e run on v0.0.18 (`/sandbox/.openclaw-data/workspace/media` symlink layer, removed entirely by #2227). + +- Set `BASELINE_INSTALL_FAILED=1` (same flag — Step 9's cap-at-84 rule keys off it regardless of which phase rotted). +- Skip 8b/8c, jump to 8d. +- Note "baseline-build-skipped" in the final comment with the specific failing layer/file so a reviewer can see *why* the v0.0.X image no longer builds (the why is usually a follow-on PR that removed the rotted layer). +- Do not retry the build with a patched Dockerfile — that breaks faithfulness. We're claiming "couldn't independently re-trigger the original symptom on baseline," not "we made the old version work somehow." + +Both baseline-rot variants share the same downstream effect: Step 9 cap, Step 10 caveat, @-mention reporter to confirm. Distinguishing them in the comment helps a reviewer understand the failure mode without re-running. + +This degradation is expected — old releases rot at multiple phases (binary installer URL drift, base-image dependencies vanish, in-image Dockerfile layers get removed by structural refactors). We still want to extract whatever signal we can from the latest run plus PR/commit evidence, just at a more conservative confidence ceiling. **Keep-box-on-inconclusive.** When `verify-inconclusive` lands (Step 8c gave up, or Step 9 score < 60), **skip the cleanup trap** for this run if the box was provisioned by this run — set `PROVISIONED_NEW=0` before the trap fires so the EXIT handler is a no-op. Print the `brev shell "$INSTANCE_NAME"` command and an explicit `brev delete "$INSTANCE_NAME"` reminder in the run output so the maintainer can triage and clean up manually. Reused boxes stay regardless. Ship-failed verifications are the exact case where having an inspectable artifact pays for itself; an unbounded sleep-and-delete in the background isn't reliable across session ends, so we leave deletion explicit. From 24fae895609242dc8503383d701255d12eb77b47 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 12:14:48 -0700 Subject: [PATCH 20/40] fix(verify-stale): make brev-login error directive, scrub stale install host MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two small UX fixes surfaced from a thorough re-read. Step 6.5's brev-auth error printed three numbered options without a clear recommendation. From a non-TTY agent harness (Claude Code or any unattended runner) the right answer is option 2 — open a separate terminal, run `brev login`, complete the browser flow, come back. The previous message buried that path inside option 2's inline comment. Replace with a directive recipe that names the recommended flow first (open separate terminal, browser auth, re-run the skill) and lists the headless alternatives below as "when option 1 isn't available." Also explicitly notes that credentials persist to ~/.brev/credentials.json, so re-running the skill picks them up automatically. Step 6.5's install-URL error still suggested checking `https://nemoclaw.nvidia.com` even after the default was switched to `https://www.nvidia.com/nemoclaw.sh` in commit 296f7cd4. Update the suggestion to match the new default and add an explicit "then re-run this skill" so the maintainer knows the flow is fix-and-retry, not abort. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 28 +++++++++++++++---- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 7540e529742..b0c21f20393 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -196,11 +196,27 @@ for cmd in gh brev jq python3 curl; do done # Brev auth — short-circuit only after the auth check, not before. +# When auth fails, give the user a directive recipe (the browser-flow path is +# what works from non-TTY harnesses like Claude Code, not the headless options). brev ls --json >/dev/null 2>&1 || { - echo "Brev not authenticated. Choose one:" - echo " 1) brev login --skip-browser # prints a URL, works from any shell" - echo " 2) brev login # opens browser, run in a separate terminal if your shell lacks a TTY" - echo " 3) brev login --token \"\$BREV_API_TOKEN\" # non-interactive, same env var used by test/e2e/brev-e2e.test.ts" + cat <<'MSG' + +ERROR: Brev not authenticated. ~/.brev/credentials.json is missing or the token expired. + +What to do (works from any harness, including non-TTY agent contexts): + + 1. Open a separate Terminal on your laptop. + 2. Run: brev login + A browser opens; complete the auth flow; the CLI exits on success. + 3. Come back here and re-run this skill. Credentials persist to + ~/.brev/credentials.json and every subsequent `brev` call picks them up. + +Headless / no-browser alternatives (when option 1 isn't available): + - brev login --skip-browser # prints a URL, paste into any browser + - brev login --token "$BREV_API_TOKEN" # non-interactive; same env var used + # by test/e2e/brev-e2e.test.ts + +MSG exit 1 } @@ -224,7 +240,9 @@ done INSTALL_URL=${NEMOCLAW_INSTALL_URL:-https://www.nvidia.com/nemoclaw.sh} curl -fsI "$INSTALL_URL" >/dev/null 2>&1 || { echo "ERROR: install URL not reachable: $INSTALL_URL" - echo "Set NEMOCLAW_INSTALL_URL or check https://nemoclaw.nvidia.com is up." + echo " - Check https://www.nvidia.com/nemoclaw.sh is up (the default Akamai-hosted entry)." + echo " - Override with NEMOCLAW_INSTALL_URL=<alternate-url> if your team mirrors the installer." + echo " - Then re-run this skill." exit 1 } ``` From 8976176c144ccf7f4498641b74983cc2c7080977 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 12:23:09 -0700 Subject: [PATCH 21/40] fix(verify-stale): land five gaps from the post-#2007 audit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five gaps verified against the source during a thorough re-read pass. The audit also caught two false-positive claims I'd made earlier (`NEMOCLAW_INSTALL_TAG` env var and `comments` field in batch fetch) that didn't survive verification — those are not landed because there was nothing to fix. Step 1 batch discovery query bumped from `--limit 100` to `--limit 1000` so the skill doesn't silently drop issues beyond the page. The recent candidate triage on this repo found 129 open bugs; the previous limit would have lost 29 of them. The 20-issue per-run processing cap is downstream of Step 3/4 filters and unaffected. Step 3's 7-day comment-marker TTL had the rule but no implementation. Add a portable `gh issue view --json comments --jq` snippet that extracts marker comments newer than the cutoff. macOS and Linux date(1) syntax differ for relative-date math, so the cutoff line tries both. Without this, the first cron run after any posting will silently re-verify everything previously marked, posting duplicate comments every Monday. Step 6.5 preconditions now checks gh identity via `gh api user --jq .login` and prints the resolved login before any later step runs. Comments posted by Step 10 land under this account; surfacing it explicitly catches the multi-token / wrong-tab / mid-session-reauth case before a public comment lands under the wrong handle. Step 10 now requires a `**Verification mode:**` header line in every template (was previously by-design only). Reader should never have to guess whether a verdict came from real install logs or from static analysis. Filled in concrete defaults for the standard fixed/inconclusive template ("runtime reproduction; baseline + latest installed and run") and the still-reproduces template ("runtime reproduction; bug confirmed live"). Step 10 also requires a link-pass self-verification on every template, not just by-design's Step 8.5d. The "Tag-anchoring + linking rule" already declared every citation must be a clickable markdown link to the verified-on tag, but the verification step that resolves those links (`gh api .../contents/<path>?ref=<tag>` or `curl -fsI`) was scoped only to the by-design path. Same 404-cite risk applies to the standard template — broken citation links advertise verification work that didn't happen, and that's worse than no citation at all. Surfaced from: end-to-end audit after the #2007 e2e run, with the specific `gh issue view`/`grep` commands run against the SKILL.md to confirm each claim before listing it as a gap. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 48 ++++++++++++++++++- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index b0c21f20393..46ebc63c1b3 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -24,10 +24,12 @@ gh issue view <number> --repo NVIDIA/NemoClaw \ --json number,title,body,labels,url,author,createdAt,comments ``` -**Batch mode** — user says "batch", "weekly", or provides no number. Cap at 20 issues per run. +**Batch mode** — user says "batch", "weekly", or provides no number. Cap at 20 issues for *processing* per run (enforced after Step 3/4 filters narrow the pool). + +The discovery query needs to see the entire open-bug pool — the per-run processing cap is downstream. Use `--limit 1000` so the skill doesn't silently drop issues beyond the page (the candidate triage run found 129 open bugs; an earlier `--limit 100` would have missed 29 of them). ```bash -gh issue list --repo NVIDIA/NemoClaw --state open --limit 100 \ +gh issue list --repo NVIDIA/NemoClaw --state open --limit 1000 \ --label bug \ --json number,title,body,labels,url,author,createdAt,comments ``` @@ -74,6 +76,31 @@ Apply these rules in order. Drop any issue that fails a rule. - The issue carries a `fixed-on-latest` or `verify-inconclusive` label. (Cleared by the release sweep in `nemoclaw-maintainer-cut-release-tag` so the issue re-opens on each release.) The by-design path uses the existing repo `status: wont-fix` label, which is already covered by the issue-type skip rule above — no separate idempotency clause needed for that path. - A comment matching `<!-- nemoclaw-verify-stale v\d+ YYYY-MM-DD -->` was posted **within the last 7 days**. The regex matches any marker version (`v1`, `v2`, …) so future skill versions can re-verify older-marked issues by tightening the regex (e.g. require a specific marker version). The marker carries a date so the candidate filter can apply a TTL — useful for the still-reproduces case (Step 9), where no label is applied and we want next week's run to re-verify rather than skip forever. +Implementation — match the marker against each comment's `createdAt`. Use `gh issue view --json comments` (single-issue mode already fetches this; batch mode's `gh issue list` also returns the comment array per issue): + +```bash +# Cutoff for the 7-day TTL. macOS and Linux date(1) syntax differ; try both. +SEVEN_DAYS_AGO=$(date -u -v-7d +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \ + || date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ) + +# Returns the timestamp of the most recent marker comment within the TTL, or empty. +RECENT_MARKER=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ + --jq --arg cutoff "$SEVEN_DAYS_AGO" ' + .comments[] + | select(.body | test("<!-- nemoclaw-verify-stale v\\d+ \\d{4}-\\d{2}-\\d{2} -->")) + | select(.createdAt > $cutoff) + | .createdAt' \ + | head -1) + +if [ -n "$RECENT_MARKER" ]; then + echo "Skip: marker posted $RECENT_MARKER (within 7-day TTL)" + # In single-issue mode: exit 0 with a friendly message. + # In batch mode: continue to the next candidate. +fi +``` + +Run this check for every candidate that survived the label-based filters above; drop those whose `RECENT_MARKER` is non-empty. + **Candidate rule:** keep the issue if **either**: - The reported version (parsed from body or labels — see Step 4) is **at least 2 versions behind** `$LATEST` in the rightmost-incrementing component, **or** @@ -195,6 +222,17 @@ for cmd in gh brev jq python3 curl; do command -v "$cmd" >/dev/null 2>&1 || { echo "ERROR: missing required dependency: $cmd"; exit 1; } done +# gh identity — every comment posted by Step 10 lands under whatever account `gh` is currently +# authenticated as. Surface that explicitly so the maintainer notices before a public comment +# lands under the wrong handle (this matters when `gh` is multi-token, after a recent re-auth, +# or when running under a service-account hostname). +GH_IDENTITY=$(gh api user --jq .login 2>/dev/null) +if [ -z "$GH_IDENTITY" ]; then + echo "ERROR: gh CLI is not authenticated. Run: gh auth login # then re-run this skill" + exit 1 +fi +echo "gh identity: @$GH_IDENTITY — comments posted by this run will appear under this handle" + # Brev auth — short-circuit only after the auth check, not before. # When auth fails, give the user a directive recipe (the browser-flow path is # what works from non-TTY harnesses like Claude Code, not the headless options). @@ -866,6 +904,10 @@ Transcripts and synth-repro scripts are already plain text and skip the pre-pass **Mandatory cap caveat.** When the score is capped (Step 9 baseline-validation gating, or any Step 11 degraded-mode path), the rendered Verdict section must include a one-line caveat naming the cap and the reason. Example: `Capped at 84 because Step 9's baseline-validation gate did not run (sandbox-build rot on v0.0.18: Dockerfile symlink layer removed by #2227).` Don't make readers reverse-engineer why the score didn't go higher — name it. +**Mandatory `Verification mode` header line.** All three templates below include a `**Verification mode:**` line in the metadata block, naming what we did and didn't actually run (e.g., "runtime reproduction on Brev <SKU>; baseline + latest both installed and run" for the standard template; "static analysis at the verified-on tag — no runtime reproduction" for the by-design template; "runtime reproduction on Brev <SKU>; bug confirmed live on latest" for still-reproduces). Reader should never have to guess whether the verdict came from real install logs or from static analysis. + +**Link-pass self-verification (all templates).** Same rule as Step 8.5d's link pass, applied to every template. Resolve at least one rendered markdown link from each section that has them (`What's structurally fixed` / `Vestigial references` / `Existing CI coverage` for by-design; `Relevant changes since` / transcript code-anchor citations for the standard template) via `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 if path exists at tag, 404 otherwise) or `curl -fsI <blob-url>`. A 404 on a citation in the rendered comment is worse than no citation — it advertises verification work that didn't actually happen. If any link fails to resolve, fix it or bail to `verify-inconclusive`. + **Mandatory closing block — reporter @-mention with confirmation language.** Every template below ends with an explicit @-mention of the original reporter using this exact shape: > @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. @@ -879,6 +921,7 @@ The skill cannot independently confirm a closed-as-fixed verdict — only the re **Reported on:** v0.0.31 **Verified on:** v0.0.34 (commit abc1234) +**Verification mode:** runtime reproduction on Brev `<instance-class>` — baseline (v0.0.31) and latest (v0.0.34) both installed and run; comparison made on the captured transcripts. (Or: "runtime reproduction on Brev `<instance-class>` — baseline-install-skipped (`.openclaw-data` rot, see Step 11), latest-only run; verdict capped at 84.") **Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 / <CUDA version if GPU> ### Baseline (reported version) @@ -931,6 +974,7 @@ The skill cannot independently confirm a closed-as-fixed verdict — only the re **Reported on:** v0.0.31 **Verified on:** v0.0.34 (commit abc1234) +**Verification mode:** runtime reproduction on Brev `<instance-class>` — baseline confirmed the symptom matches the issue; latest (v0.0.34) also produced the symptom. Bug is still live. **Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 The skill ran the reported reproducer on v0.0.34 and observed the same bug symptom described in this issue. The bug is still live. From 9e047012cd33a329e064ed6e8a9d30dcf741838e Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 12:27:01 -0700 Subject: [PATCH 22/40] fix(verify-stale): drop batch cap to 15 and actually enforce it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes that together turn the per-run batch cap from policy into code. Lower the per-batch processing cap from 20 to 15. Sequential execution with 1-2 reused Brev boxes works fine for runs in this size range; ~2-3 hours wallclock for a full 15-issue batch is the comfortable budget before either the per-plan approval gate breaks down or the maintainer session needs to span multiple sittings. Add the slicing code that actually enforces the cap. Previously Step 1 declared "Cap at N issues per run" as policy and nothing applied it — candidate sets larger than the cap would just process to completion silently. Move the slice to the end of Step 4 (after Step 3 label filters and the Step 4 version+candidate-rule filters narrow the pool) and sort by `(-versions_behind, -age_days)` so the most stale come first. Spillover beyond 15 stays eligible for the next run via the marker-comment 7-day TTL added in 8976176c. Single-issue mode bypasses the cap entirely; the maintainer named the issue explicitly. Cadence section updated from "≤20 issues" to "≤15 issues" to match. Surfaced from the post-#2007 audit: of the 13 gaps initially called out, the cap-enforcement one was operational toil (skill could chew through cost on a runaway batch). Lowering and enforcing closes that toil path. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 46ebc63c1b3..9c8486dc49a 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -24,7 +24,7 @@ gh issue view <number> --repo NVIDIA/NemoClaw \ --json number,title,body,labels,url,author,createdAt,comments ``` -**Batch mode** — user says "batch", "weekly", or provides no number. Cap at 20 issues for *processing* per run (enforced after Step 3/4 filters narrow the pool). +**Batch mode** — user says "batch", "weekly", or provides no number. Cap at **15 issues** for *processing* per run, enforced as a slice after Step 3/4 filters narrow the pool. The cap exists because batch is sequential (Step 7 reuse-or-provision keeps it on 1–2 Brev boxes total) and the wallclock budget is ~2–3 hours per 15-issue run; running larger forces the maintainer to either drop the per-plan approval gate or spread the batch across multiple sessions. The discovery query needs to see the entire open-bug pool — the per-run processing cap is downstream. Use `--limit 1000` so the skill doesn't silently drop issues beyond the page (the candidate triage run found 129 open bugs; an earlier `--limit 100` would have missed 29 of them). @@ -142,6 +142,21 @@ If no version survives, drop the issue from the candidate set — we cannot esta **Variable format for downstream steps.** Set `REPORTED_VERSION` to the **full tag string** (e.g., `REPORTED_VERSION="v0.0.32"`), not just the patch number. Step 8a's installer expects the full tag via the `NEMOCLAW_INSTALL_TAG` env var. +**Batch cap enforcement.** In batch mode, after Step 3 label filters and the Step 4 version+candidate-rule filters narrow the pool, sort surviving candidates by `(-versions_behind, -age_days)` so the most stale come first, then **slice to the top 15**: + +```bash +# Each candidate has at minimum: number, reported, behind, age_days +SLICED=$(printf '%s' "$CANDIDATES_JSON" | jq ' + sort_by([-(.behind // 0), -(.age_days // 0)]) + | .[0:15]') +SLICED_COUNT=$(printf '%s' "$SLICED" | jq 'length') +TOTAL=$(printf '%s' "$CANDIDATES_JSON" | jq 'length') +echo "Batch run: processing $SLICED_COUNT of $TOTAL eligible candidates (cap: 15)." +[ "$TOTAL" -gt 15 ] && echo " Spillover: $((TOTAL - 15)) candidates deferred to next run; the marker-comment TTL (Step 3) keeps them eligible." +``` + +The slice is the only enforcement of the cap — without it, "Cap at 15" is policy that nothing actually applies. Single-issue mode bypasses the cap entirely (the user explicitly named one issue). + **NVBugs cross-reference.** Many NV QA bugs include an NVBugs ticket footer like `[NVB#6100043]`. Extract it at the same time as the version so Step 8.5's comment template (and any other comment template that wants to mention it) can include the cross-reference: ```bash @@ -1104,7 +1119,7 @@ Never stage or commit the log to the NemoClaw repo. ## Cadence -- **Weekly cron** — Monday morning, batch mode, ≤20 issues. +- **Weekly cron** — Monday morning, batch mode, ≤15 issues (the Step 1 cap, sliced after Step 3/4 filters). - **Manual** — invoke with a single issue number anytime. --- From 94d444bdcb37308e456f098adb70c352a776aa23 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 12:36:44 -0700 Subject: [PATCH 23/40] fix(verify-stale): land six bug-shape gaps from the candidate-set audit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Looked at the 24 candidates surfaced by the recent triage run by *kind* of bug rather than by version, and found six gaps where the standard rubric would produce a wrong verdict or hang. All six landed in this commit. Step 3 platform skip: drop `Platform: Jetson AGX Thor/Orin`. Brev has no equivalent silicon (embedded/edge ARM with integrated GPU is not in the SKU catalog), so any Brev verification of a Jetson-only bug produces a misleading "fixed-on-x86" verdict. `Platform: DGX Spark` and `Platform: GB10` stay in scope but Step 10 now requires a hardware-substitution caveat naming the Brev SKU we substituted. Step 3 TUI / interactive-UI skip: drop issues whose title contains TUI / dashboard UI / chat UI / keystroke / key press, or whose body describes interactive UI behavior without a non-interactive reproducer. `brev exec` does not allocate a real TTY by default; TUI reproducers hang or silently fail at the first prompt. v1 documents this as out-of-scope; v1.1 may add a script(1) / expect / tmux harness. Step 5 now classifies bug-class in addition to CPU/GPU. Four classes: `performance`, `rebuild-cycle`, `log-only`, `functional` (default). Detection heuristics from the issue body (latency thresholds, "across rebuilds" / "after restart", "see lots of error in <X> log"). Each class routes to a different Step 8 rubric. Step 8b: log-scraping. When `BUG_CLASS=log-only`, also pull `~/.openclaw/logs/*.log` and `/var/log/nemoclaw/*.log` from inside the sandbox after the reproducer runs and search them for the issue's symptom phrase. Some bugs describe symptoms in internal log files, not the reproducer's stdout; the previous match rubric only checked the transcript. Step 8b: flake-detection retry. For functional bugs, run baseline three times if the first run shows the symptom inconsistently. Mixed results (1 or 2 of 3 reproduce) trigger a "flake suspected" caveat, −25 score adjustment, and downgrade `+50 latest clean` to `+25` so a lucky-clean-latest-run on an intermittent bug doesn't silently become a fixed-on-latest verdict. New Step 8e: performance-bug verification. Multi-run latency distribution rubric — N=10 runs each side, compute p50/p90, parse SLA from issue body, match latest's distribution against the SLA rather than against a symptom phrase. Cap at 60 unless the bug is silicon-independent because Brev SKUs aren't faithful to DGX Spark or GB10 hardware for performance-shape bugs. New Step 8f: rebuild-cycle verification. Run-rebuild-rerun harness for bugs that only manifest across destroy/recreate boundaries (#2701-shape). Capture artifacts pre-rebuild, trigger `nemoclaw destroy --all --force && nemoclaw onboard`, re-capture post-rebuild, diff to determine whether the artifact persisted as the issue expects. Step 10 mandatory hardware-substitution caveat: when DGX Spark or GB10 is in the issue's labels and Step 7 substituted with a different silicon class, the comment metadata block must name the substitution explicitly so silicon-shape bugs get the right reader expectation. Surfaced from: stress-testing the skill against the actual shapes of bugs in the 24-candidate set, not just against the bugs we already verified. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 90 ++++++++++++++++++- 1 file changed, 89 insertions(+), 1 deletion(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 9c8486dc49a..d73f6832cb8 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -65,7 +65,9 @@ Apply these rules in order. Drop any issue that fails a rule. **Issue-type allowlist:** must have `bug` label. **Issue-type skip:** drop if any of `enhancement`, `documentation`, `status: wont-fix`, `status: needs-info`, `security`. Use the canonical repo label names — bare `wontfix` / `needs-info` are NOT the repo's labels (verified via `gh label list`); the actual labels carry a `status:` prefix and a hyphen. -**Platform skip (Linux-only in v1):** drop if any of `Platform: Windows/WSL`, `Platform: MacOS`, `Platform: macOS`. Keep `Platform: Ubuntu`, `Platform: DGX Spark`, `Platform: GB10`, `Platform: All`, or no platform label. +**Platform skip (Brev-reproducible only in v1):** drop if any of `Platform: Windows/WSL`, `Platform: MacOS`, `Platform: macOS`, `Platform: Jetson AGX Thor/Orin`. Brev has no equivalent hardware for Jetson (embedded/edge ARM with integrated GPU is not in the Brev SKU catalog), so any Brev verification of a Jetson-only bug would produce a misleading "fixed-on-x86" verdict. Keep `Platform: Ubuntu`, `Platform: DGX Spark`, `Platform: GB10`, `Platform: All`, or no platform label. `Platform: DGX Spark` and `Platform: GB10` stay in scope but Step 10 requires a "Hardware substitution" caveat in the comment naming the Brev SKU we used as a substitute (Brev x86 GPU SKUs are not faithful to GB10 / Grace Hopper silicon for performance-shape or memory-architecture-shape bugs). + +**TUI / interactive-UI skip:** drop if the issue title contains `TUI`, `dashboard UI`, `chat UI`, `keystroke`, or `key press`, OR if the body describes interactive UI behavior (key sequences, mouse interactions, browser-side UI state) without a non-interactive reproducer (no `NEMOCLAW_NON_INTERACTIVE=1` or equivalent env var pattern). `brev exec` does not allocate a real TTY by default, so TUI reproducers hang or silently fail at the first prompt; v1 documents this as out-of-scope rather than emitting a wrong verdict. v1.1 may add a `script(1)` / `expect` / `tmux send-keys` harness to lift this skip. **Integration skip (deferred to v2):** drop if any of `Integration: Slack`, `Integration: Discord`, `Integration: Telegram`, `Integration: Hermes`, `Integration: OpenClaw`, `Integration: WeChat`. These need third-party credentials a fresh Brev box cannot provide. @@ -193,6 +195,17 @@ Three real failure modes surfaced during the v1 dry-run. Test each before trusti CPU default keeps cost low. Only escalate to GPU when the reproducer needs one. +**Bug class classification.** In addition to CPU/GPU, classify the bug's verification shape so Step 8 routes to the right rubric. Classes are mutually exclusive — pick the first that matches: + +| Class | Detection heuristic | Routes to | +|---|---|---| +| `performance` | Body or title mentions latency thresholds (`P50`, `P90`, `ms`, `seconds`, `slow`, `hangs`, `timeout` with a numeric value), or mentions `memory leak` / `over time` / `eventually` | Step 8e (multi-run distribution rubric) | +| `rebuild-cycle` | Body mentions `rebuild`, `recreate`, `restart`, `pod recreate`, `across rebuilds`, `after restart`, `survives a destroy` | Step 8f (run-rebuild-rerun harness) | +| `log-only` | Body's symptom is logs-not-stdout: `see lots of error in <X> log`, `os.networkInterfaces guard errors`, anything pointing at a specific log file rather than the reproducer's stdout/stderr | Step 8b's match rubric extended with log-scraping | +| `functional` (default) | Everything else — exit code + stdout/stderr matching | Step 8b standard rubric | + +Most bugs are `functional`. The other three classes need verification harnesses that the standard rubric can't produce honestly — e.g., one clean run of a perf reproducer doesn't tell you the p50 budget was met; one onboard run doesn't tell you a config survives a rebuild. Set `BUG_CLASS=<class>` so downstream steps can branch. + --- ## Step 6: Extract the Reproducer @@ -545,6 +558,29 @@ brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcript.log ``` +**Log-scraping (when `BUG_CLASS=log-only`).** Some bugs describe symptoms that show up in internal log files, not the reproducer's stdout/stderr — e.g., #1642 "see lots of error in openclaw log," #2611 "os.networkInterfaces guard errors." After running the reproducer, also pull the relevant logs from inside the sandbox and search them for the issue's symptom phrase: + +```bash +# Common NemoClaw / OpenClaw / OpenShell log paths inside the sandbox. +brev exec "$INSTANCE_NAME" "sg docker -c 'cat ~/.openclaw/logs/*.log /var/log/nemoclaw/*.log 2>/dev/null'" \ + | tee ./baseline-logs.log + +# Search the log capture for the issue's symptom phrase too, not just the transcript. +grep -F "<symptom phrase from issue body>" ./baseline-logs.log +``` + +For functional bugs the reproducer's stdout is sufficient; for log-only bugs the transcript may be clean but the log capture has the symptom. Both halves feed into the match rubric below. + +**Flake-detection retry.** Even for `functional` bugs, race-prone reproducers (TUI rendering, network policy negotiation, concurrent sandbox state) can produce inconsistent results. Run baseline three times if the first run shows the symptom inconsistently — same script, same env, just three back-to-back invocations. If the three runs disagree, that's signal: + +| 3-run baseline result | Verdict | +|---|---| +| All three reproduce the symptom | Strong baseline match → continue to 8d | +| All three are clean (no symptom) | Reproducer doesn't expose the bug on baseline → Step 8c synth-repro | +| Mixed (1 or 2 of 3 show the symptom) | Flake-prone reproducer. Note "flake suspected" in the comment; apply −25 to Step 9 score; downgrade `+50 latest clean` to `+25` because a clean latest run could just be the lucky path of an intermittent bug | + +Skip flake retry for `performance` and `rebuild-cycle` classes — those have their own multi-run rubrics in Steps 8e and 8f. + **Match rubric.** LLM compares `baseline-transcript.log` to the issue's "Actual result" / error description. Match criteria, in order: 1. **Exit code agrees** with what the issue describes (non-zero if issue describes a failure, zero if issue describes a wrong-output bug). Necessary but not sufficient. @@ -637,6 +673,56 @@ Adapt the axes to the bug class. For filesystem bugs: `find`, `lsattr`, `stat`. --- +## Step 8e: Performance-Bug Verification (when `BUG_CLASS=performance`) + +Performance bugs (#2598 "10s P50", #2600 "hangs ~2 min", #2733 Ollama tool-call leak over time) can't be answered by the standard exit-code + symptom-phrase rubric — one clean reproducer run doesn't tell you the p50 budget is met; one slow run doesn't tell you the bug still reproduces. Replace Step 8b's match with a measurement-and-distribution rubric: + +1. **Parse the SLA from the issue body.** Extract numeric latency thresholds: `10s P50`, `200ms`, `under 5 seconds`, `~2 min`. Save as `SLA_P50_MS`, `SLA_P90_MS`, etc. If no numeric SLA is in the body, route to Step 8c synth-repro to ask the reporter (via comment) for one — without a target, the verdict is undefined. +2. **Run the reproducer N=10 times** on each side (baseline + latest), capturing per-run latency: + + ```bash + for i in $(seq 1 10); do + /usr/bin/time -f '%e' bash ~/reproducer.sh >/dev/null 2>>./latest-perf.log + done + ``` + +3. **Compute p50 and p90** for both sides. `sort -n ./latest-perf.log | awk 'NR==5'` for p50 of 10 runs. +4. **Match rubric:** + - Latest's p50 within the SLA AND baseline's p50 outside the SLA → bug fixed; same Step 9 scoring (subject to baseline-validation gate). + - Latest's p50 outside the SLA → bug still reproduces (Step 9 special case). + - Latest p50 within SLA AND baseline p50 also within SLA → reproducer doesn't actually exercise the bug; route to Step 8c synth-repro. + +**Hardware-substitution caveat.** Performance numbers are silicon-dependent. When the issue is `Platform: DGX Spark` or `Platform: GB10` and we're measuring on a Brev x86 GPU SKU, the comment must say so explicitly: a Brev p50 of 1.5s on a `H100` does not prove the DGX Spark p50 is fixed. Cap the score at 60 unless the bug is clearly silicon-independent (e.g. an algorithmic regression in user-space JS that would manifest the same on any silicon). + +--- + +## Step 8f: Rebuild-Cycle Verification (when `BUG_CLASS=rebuild-cycle`) + +Rebuild-cycle bugs (#2701 "Pod recreate wipes `/tmp/nemoclaw-proxy-env.sh`," issues describing "configuration is not persisted across rebuilds") only manifest when sandbox state crosses a destroy/recreate boundary. A single onboard run can't trigger the symptom. Replace Step 8b's match with a run-rebuild-rerun harness: + +1. **First onboard.** Run the reproducer once to establish initial state. Capture relevant artifacts (config files, env vars, sandbox metadata) — the issue body usually names what should persist: + + ```bash + brev exec "$INSTANCE_NAME" "sg docker -c 'cat <files-mentioned-in-issue> 2>&1'" | tee ./pre-rebuild.log + ``` + +2. **Trigger the rebuild.** Use `nemoclaw destroy --all --force` followed by `nemoclaw onboard` with the same env vars. Do NOT comprehensive-reset between (the point is to test the destroy/recreate, not start from scratch). + +3. **Re-capture the same artifacts** post-rebuild: + + ```bash + brev exec "$INSTANCE_NAME" "sg docker -c 'cat <same-files> 2>&1'" | tee ./post-rebuild.log + ``` + +4. **Diff and match.** The bug is "X gets wiped / changes / regresses across rebuild." Compare pre-rebuild vs post-rebuild captures to the issue's expected behavior: + - Pre and post agree (artifact preserved) AND issue says it should be preserved → bug fixed + - Pre and post differ (artifact wiped) AND issue says it gets wiped → bug still reproduces + - Pre and post agree AND issue says it gets wiped → reproducer doesn't exercise the bug; Step 8c synth-repro + +The harness still uses Step 9's scoring framework — `+50 latest clean (artifact preserved)`, etc. — but the "what gets compared" axis is the diff, not the symptom phrase. + +--- + ## Step 8.5: Detect "Behavior Changed by Design" Before scoring, check whether the symptom is intentional. Some bugs are filed against behavior that was **deliberately changed or removed** in a merged PR — running the standard rubric on these produces misleading verdicts. The symptom "still reproduces" but the right answer is "won't fix, see PR #X." Issue #2791 is the prototype: `config set` was removed in PR #2227, the reporter tested a version that already had it gone, and a standard rubric run would have buried that context under a low-confidence `verify-inconclusive` label. @@ -919,6 +1005,8 @@ Transcripts and synth-repro scripts are already plain text and skip the pre-pass **Mandatory cap caveat.** When the score is capped (Step 9 baseline-validation gating, or any Step 11 degraded-mode path), the rendered Verdict section must include a one-line caveat naming the cap and the reason. Example: `Capped at 84 because Step 9's baseline-validation gate did not run (sandbox-build rot on v0.0.18: Dockerfile symlink layer removed by #2227).` Don't make readers reverse-engineer why the score didn't go higher — name it. +**Mandatory hardware-substitution caveat.** When the issue carries `Platform: DGX Spark` or `Platform: GB10` and Step 7 provisioned a Brev SKU that is not the same silicon (Brev's stoppable GPU catalog is x86 + discrete H100/A100/L40S/T4 — not Grace Hopper / GB10 unified-memory ARM64), the rendered comment must include a one-line "Hardware substitution" note. Example: `Hardware substitution: verified on Brev n1-standard-4:nvidia-tesla-t4 (x86_64 + T4) as a substitute for the reporter's DGX Spark (ARM64 + GB10). For silicon-shape bugs (perf, memory architecture, drivers) this is not a faithful repro — please confirm on actual DGX Spark.` This goes in the metadata block right after `Verification mode:` so it's visible at the top, not buried in the analysis. + **Mandatory `Verification mode` header line.** All three templates below include a `**Verification mode:**` line in the metadata block, naming what we did and didn't actually run (e.g., "runtime reproduction on Brev <SKU>; baseline + latest both installed and run" for the standard template; "static analysis at the verified-on tag — no runtime reproduction" for the by-design template; "runtime reproduction on Brev <SKU>; bug confirmed live on latest" for still-reproduces). Reader should never have to guess whether the verdict came from real install logs or from static analysis. **Link-pass self-verification (all templates).** Same rule as Step 8.5d's link pass, applied to every template. Resolve at least one rendered markdown link from each section that has them (`What's structurally fixed` / `Vestigial references` / `Existing CI coverage` for by-design; `Relevant changes since` / transcript code-anchor citations for the standard template) via `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 if path exists at tag, 404 otherwise) or `curl -fsI <blob-url>`. A 404 on a citation in the rendered comment is worse than no citation — it advertises verification work that didn't actually happen. If any link fails to resolve, fix it or bail to `verify-inconclusive`. From 1d625dd84d2d75b9e08af7761652b3828e1ca76b Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 14:03:15 -0700 Subject: [PATCH 24/40] fix(verify-stale): land six gaps from the post-#2592 audit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six gaps surfaced from the second e2e run (issue #2592) and the broader audit of how the skill handles provider classification. Step 5 now classifies provider in addition to CPU/GPU and bug-class. Detection signals from labels and body keywords identify NIM, Gemini, Anthropic, Bedrock, or Ollama. When the reproducer references a non-Ollama provider AND actually exercises inference, the skill stops at Step 5 and prompts the maintainer interactively before any Brev cost — three options: provide the API key, accept Ollama substitution with the -30 penalty, or skip to verify-inconclusive. Pure-CLI / pure-sandbox bugs are exempt because the provider doesn't matter. Step 6 reproducer-extraction regex extended to also match `openclaw` and `openshell` invocations as anchor words. Issue #2592's reproducer was `openclaw channels add telegram` run inside the sandbox; the previous `nemoclaw`-only regex would have missed the verbatim block. Step 8a now passes the provider env vars through to install.sh's bundled onboard step, so it doesn't fall back to the default `build` (NIM) provider and fail with the misleading NVIDIA_API_KEY missing error. NEMOCLAW_PROVIDER=ollama (the default case) makes the bundled onboard use the local Ollama set up in Step 8a.5; NVIDIA_API_KEY only flows through if the maintainer provided one at Step 5's prompt. The bundled onboard creates a throwaway sandbox that gets destroyed before the reproducer runs. The reproducer's own onboard should pass `--fresh` so a half-built install-sandbox doesn't trip the "previous session failed" guard. Step 8a.5 now has an Ollama-coverage table making explicit which bug classes Ollama covers faithfully (CLI, sandbox, networking) vs which ones it doesn't (provider-specific behavior, model-specific behavior, silicon-dependent perf). Step 5's prompt keys off this table. Step 8a.5b documents the openshell-sandbox-exec syntax footgun. The correct form is `openshell sandbox exec -n NAME -- CMD`; the wrong form silently auto-detects the sandbox by "last used" and stuffs the leftover positional into bash's $0. Same section gets a brev-exec re-execution guard via a sentinel file at ~/.verify-stale-running to prevent the double-onboard scenario when SSH drops mid-run. Step 11 reframes baseline-build-rot as the dominant failure mode for any reported version >5-7 patches behind, not an edge case. Both Brev e2e runs hit it. Cap-at-84 with reporter @-mention is the modal verdict shape, not the exception — pre-flight carries more weight than baseline runtime evidence for older bugs. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 124 +++++++++++++++++- 1 file changed, 117 insertions(+), 7 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index d73f6832cb8..5c25b4356ad 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -206,6 +206,41 @@ CPU default keeps cost low. Only escalate to GPU when the reproducer needs one. Most bugs are `functional`. The other three classes need verification harnesses that the standard rubric can't produce honestly — e.g., one clean run of a perf reproducer doesn't tell you the p50 budget was met; one onboard run doesn't tell you a config survives a rebuild. Set `BUG_CLASS=<class>` so downstream steps can branch. +**Provider classification.** Some bugs are tied to a specific inference provider (NVIDIA NIM, Gemini, Anthropic, OpenAI) and won't reproduce faithfully under Ollama substitution. Classify which provider the issue references so downstream steps either prompt for the right API key or accept the substitution penalty: + +| Detection signal | Provider | +|---|---| +| `Provider: NVIDIA` label, body mentions `NVIDIA NIM`, `build.nvidia.com`, `nvapi-...`, `NVIDIA_API_KEY`, or `NEMOCLAW_PROVIDER=build` | `nim` | +| `Provider: Gemini` label, body mentions `Gemini`, `gemini-flash`, `gemini-pro`, `GEMINI_API_KEY` | `gemini` | +| `Provider: Anthropic` / `Provider: AWS` (Bedrock) labels or matching keywords | `anthropic`/`bedrock` | +| `Provider: Ollama`, body mentions `ollama` or `NEMOCLAW_PROVIDER=ollama`, or no provider mentioned at all | `ollama` (default) | + +Set `BUG_PROVIDER=<provider>`. + +**Required-API-key prompt.** When `BUG_PROVIDER` is anything other than `ollama` AND the bug's reproducer actually exercises inference (not pure CLI surface or sandbox build), the skill MUST stop here and prompt the maintainer interactively before any Brev cost is incurred: + +```text +The reporter's reproducer uses the <provider> provider, which requires a real API key +to verify faithfully. Three options: + + 1. Provide an API key now. Export NVIDIA_API_KEY=<key> (or GEMINI_API_KEY=<key>, etc.) + in the environment running this skill, then re-run. The key is propagated to the + Brev box via `brev exec` and removed when the box is deleted. + + 2. Substitute Ollama and accept the -30 confidence penalty (per Step 8a.5). The + verdict will be capped because we're not exercising the real provider's code + path. + + 3. Skip this issue. Mark `verify-inconclusive` with the reason "requires <provider> + API key — not provided in this run." + +Choose 1, 2, or 3: +``` + +This prompt blocks before Step 7 provisions a box. Don't burn cost on a verification path the maintainer hasn't agreed to. + +**Pure-CLI / pure-sandbox-build bugs are exempt** — those don't actually exercise inference, so the provider doesn't matter even if the issue body mentions one. Heuristic: if Step 6.7's local-first predicate would have fired (no sandbox state, no model server interaction), skip the prompt. + --- ## Step 6: Extract the Reproducer @@ -217,7 +252,7 @@ NV QA files most bugs through an HTML form, so issue bodies are typically a mix 1. **Verbatim:** the first markdown fence (```` ``` ```` or ```` ~~~ ````) **or** HTML `<pre>` block containing a `nemoclaw` invocation. Strip surrounding tags and unescape HTML entities before saving to `./reproducer.sh`. No confidence penalty (yet). 2. **No verbatim block found:** leave `./reproducer.sh` absent. Step 8b will synthesize from the issue body on demand and apply the **−30 synth penalty** at that point. -A robust extractor handles both shapes with the body fetched as JSON: +A robust extractor handles both shapes with the body fetched as JSON. The "anchor word" — what marks a block as a reproducer — must include `nemoclaw`, `openclaw`, AND `openshell`. Issue #2592 surfaced this gap: its reproducer was `openclaw channels add telegram` run inside the sandbox; a `nemoclaw`-only regex would have missed the verbatim block and forced the run through Step 8c synth-repro with a -30 penalty: ```bash BODY=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json body -q .body) @@ -225,9 +260,12 @@ BODY=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json body -q .body REPRODUCER=$(printf '%s' "$BODY" | python3 -c ' import re, sys, html b = sys.stdin.read() -m = re.search(r"```(?:bash|sh)?\n(.*?nemoclaw.*?)\n```", b, re.S) -if not m: m = re.search(r"~~~(?:bash|sh)?\n(.*?nemoclaw.*?)\n~~~", b, re.S) -if not m: m = re.search(r"<pre[^>]*>(.*?nemoclaw.*?)</pre>", b, re.S) +# Anchor word: any of nemoclaw / openclaw / openshell. Issue bodies use whichever +# tool the reporter ran (host-side nemoclaw vs in-sandbox openclaw vs openshell CLI). +ANCHOR = r"(?:nemoclaw|openclaw|openshell)" +m = re.search(rf"```(?:bash|sh)?\n(.*?{ANCHOR}.*?)\n```", b, re.S) +if not m: m = re.search(rf"~~~(?:bash|sh)?\n(.*?{ANCHOR}.*?)\n~~~", b, re.S) +if not m: m = re.search(rf"<pre[^>]*>(.*?{ANCHOR}.*?)</pre>", b, re.S) if m: text = re.sub(r"<[^>]+>", "", m.group(1)) print(html.unescape(text).strip()) @@ -471,12 +509,32 @@ The installer accepts the target ref via the `NEMOCLAW_INSTALL_TAG` env var (ver ```bash brev exec "$INSTANCE_NAME" "$RESET" -brev exec "$INSTANCE_NAME" "NEMOCLAW_INSTALL_TAG=$REPORTED_VERSION bash -c 'curl -fsSL $INSTALL_URL | bash'" \ - || BASELINE_INSTALL_FAILED=1 +# Pass the provider env vars through so install.sh's bundled `[3/3] Onboarding` step +# doesn't fall back to the default `build` (NIM) provider — which requires NVIDIA_API_KEY +# and otherwise fails the install with a misleading error. When NEMOCLAW_PROVIDER=ollama +# (the common case), the bundled onboard uses the local Ollama we set up in Step 8a.5 +# and either succeeds (ideal) or fails on a real Dockerfile/sandbox-build issue (which +# is what we want to detect). Pass NVIDIA_API_KEY only if the maintainer provided one +# at Step 5's prompt. +brev exec "$INSTANCE_NAME" " + NEMOCLAW_INSTALL_TAG=$REPORTED_VERSION \ + NEMOCLAW_NON_INTERACTIVE=1 \ + NEMOCLAW_PROVIDER=${NEMOCLAW_PROVIDER:-ollama} \ + NEMOCLAW_MODEL=${NEMOCLAW_MODEL:-nemotron-3-nano:4b} \ + NEMOCLAW_SANDBOX_NAME=__verify_stale_install__ \ + ${NVIDIA_API_KEY:+NVIDIA_API_KEY=$NVIDIA_API_KEY} \ + bash -c 'curl -fsSL $INSTALL_URL | bash' +" || BASELINE_INSTALL_FAILED=1 brev exec "$INSTANCE_NAME" "nemoclaw --version" + +# The bundled onboard creates a sandbox named __verify_stale_install__ that we don't want. +# Destroy it so the reproducer starts from a clean state. +brev exec "$INSTANCE_NAME" "sg docker -c 'nemoclaw destroy --all --force 2>/dev/null || true'" ``` -If install fails (old releases rot — installer URLs, deps, OS images all drift over time), set `BASELINE_INSTALL_FAILED=1` and **skip 8b/8c**, going straight to 8d. Note "baseline-install-skipped" in the final comment. Step 9's scoring rule handles the degraded mode. +If install fails (old releases rot — installer URLs, deps, OS images all drift over time, or the in-image Dockerfile patch step asserts against a code shape that's since changed), set `BASELINE_INSTALL_FAILED=1` and **skip 8b/8c**, going straight to 8d. Note "baseline-install-skipped" or "baseline-build-skipped" in the final comment depending on which phase rotted. Step 9's scoring rule handles the degraded mode (cap at 84). + +**The reproducer's own `nemoclaw onboard` (Step 8b) must pass `--fresh`.** If install.sh's bundled onboard was in an in-progress or failed state when we destroyed the install sandbox, the reproducer's onboard would error with `Previous onboarding session failed. Re-run with --fresh to discard it`. `--fresh` ensures a clean start. ### Step 8a.5: Bootstrap reproducer dependencies @@ -516,6 +574,22 @@ Bootstrap **once before Step 8b's baseline run** and reuse for Step 8d's latest **If bootstrap fails** (network issue pulling the model, service won't start, etc.), this is an infra failure — abort to Step 11. Do not silently substitute; the user opted into faithfulness for a reason. +**Ollama coverage table.** Ollama is the default provider for verification runs because it's free, local, and self-hosted. It covers most bug classes faithfully but not all. Use this table to decide whether Ollama is sufficient or whether Step 5's API-key prompt should fire: + +| Bug class | Ollama covers? | Notes | +|---|---|---| +| CLI surface (subcommand parsing, flag handling, oclif dispatch) | ✓ Always | Provider not exercised | +| Sandbox structure (build, file permissions, mounts, layout) | ✓ Always | Provider not exercised | +| Networking / policy (port forwards, NAT, egress rules, channels guards) | ✓ Always | Provider not exercised | +| Generic inference flow (does an agent turn complete, does the proxy route correctly) | ✓ Usually | Ollama can fail in the same shape as NIM/Gemini for most flow bugs | +| Provider-specific behavior (`Provider: NVIDIA` symptom, NIM-only error handling, `Provider: Gemini` quirks) | ✗ No | Different code paths; substitution doesn't exercise the bug | +| Model-specific behavior (`gemini-flash-3-preview` doesn't handle prompt X, `nemotron-3-nano:4b` works fine) | ✗ No | Wrong model = wrong outputs | +| Ollama-shape-specific (#2519 "Ollama-local 401" — local-vs-networked Ollama config) | △ Sometimes | A generic Ollama install may or may not reproduce; may need specific configuration | +| Performance / latency on specific silicon | ✗ No | Hardware substitution caveat (Step 10) and Step 8e perf rubric apply | +| Quota / rate-limit / API-key validation | ✗ No | Ollama doesn't have those failure modes | + +When the table says ✗ No or △ Sometimes, Step 5's API-key prompt fires. When it says ✓, proceed with Ollama and skip the prompt. + ### Step 8a.5b: Brev exec environment quirks Two non-obvious gotchas surfaced during the #2007 e2e run that every subsequent `brev exec` call has to handle. Encode them once here so reproducer scripts don't have to relearn each time. @@ -539,6 +613,36 @@ brev exec "$INSTANCE" "sg docker -c 'bash ~/reproducer.sh'" Both patterns appear in the canonical setup script committed alongside the skill (or are encoded in your reproducer wrapper). Don't rely on the user discovering them mid-run. +**`openshell sandbox exec` argument-order footgun.** When the reproducer needs to run a command *inside* the sandbox (channels-guard checks, in-sandbox file inspection, etc.), the correct non-interactive form uses `-n <name>` and a `--` separator: + +```bash +# Correct: +openshell sandbox exec -n ai -- bash -c 'source /sandbox/.bashrc; openclaw channels add telegram; echo "EXIT=$?"' + +# Wrong (silently auto-detects sandbox by "last used", stuffs the leftover positional +# `ai` into bash's $0, prints "/bin/bash: line 1: ai: command not found" — the +# reproducer appears to fail but actually never ran inside the sandbox at all): +openshell sandbox exec ai bash -c '...' +``` + +Issue #2592's first run hit this — wasted ~15 min before the maintainer noticed. Always use the `-n <name> -- <cmd>` form when the reproducer touches in-sandbox commands. + +**`brev exec` SSH-drop re-execution guard.** Brev's CLI silently retries from the top when the SSH connection drops mid-run, producing two parallel reproducer executions (we hit this on #2592 — one onboard process clobbered another's state, and both got billed). Use a sentinel file in the reproducer wrapper to make the script idempotent: + +```bash +# At the top of the reproducer wrapper script: +SENTINEL=~/.verify-stale-running +if [ -f "$SENTINEL" ]; then + echo "ERROR: another verify-stale run is in progress (sentinel: $SENTINEL)." + echo " If you're sure no other run is active, rm $SENTINEL and re-invoke." + exit 1 +fi +trap 'rm -f "$SENTINEL"' EXIT +touch "$SENTINEL" +``` + +The sentinel survives an SSH drop because it lives on the Brev box's filesystem; the trap removes it on script exit. A second `brev exec` invocation that tries to retry from the top will hit the sentinel and bail instead of double-running. + --- ### Step 8b: Run reproducer on baseline, compare to issue symptom @@ -1148,6 +1252,12 @@ Both baseline-rot variants share the same downstream effect: Step 9 cap, Step 10 This degradation is expected — old releases rot at multiple phases (binary installer URL drift, base-image dependencies vanish, in-image Dockerfile layers get removed by structural refactors). We still want to extract whatever signal we can from the latest run plus PR/commit evidence, just at a more conservative confidence ceiling. +**Empirical reality after two e2e runs:** baseline-build-rot is the **dominant** failure mode for any reported version more than ~5–7 patches behind, not an edge case. Both #2007 (v0.0.18, 17 patches behind) and #2592 (v0.0.28, 7 patches behind) hit it. The cap-at-84 with reporter @-mention is the **modal** verdict shape for stale-issue verification, not the exception. Reframe expectations accordingly: + +- For issues reported >5 patches behind `$LATEST`, plan for the cap-at-84 path. Pre-flight (PR-search, pickaxe) carries more weight than baseline runtime evidence. +- For issues reported within 1–4 patches of `$LATEST`, baseline is more likely to install cleanly and the full +50 path is reachable. +- The skill's design assumes baseline + latest both run cleanly; in practice latest-only with cap-at-84 is the workhorse path. The score-cap is doing real work, not just a fallback. + **Keep-box-on-inconclusive.** When `verify-inconclusive` lands (Step 8c gave up, or Step 9 score < 60), **skip the cleanup trap** for this run if the box was provisioned by this run — set `PROVISIONED_NEW=0` before the trap fires so the EXIT handler is a no-op. Print the `brev shell "$INSTANCE_NAME"` command and an explicit `brev delete "$INSTANCE_NAME"` reminder in the run output so the maintainer can triage and clean up manually. Reused boxes stay regardless. Ship-failed verifications are the exact case where having an inspectable artifact pays for itself; an unbounded sleep-and-delete in the background isn't reliable across session ends, so we leave deletion explicit. --- From c6e65081a741a821e8e291974575a69be9364e9c Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 15:47:35 -0700 Subject: [PATCH 25/40] fix(verify-stale): echo resolved nemoclaw version after install, fix sandbox name MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two fixes from a rot-debugging investigation that nearly produced a wrong conclusion. When testing whether old NemoClaw versions actually rot under the literal end-user invocation (`https://www.nvidia.com/nemoclaw.sh`), the first test run was botched by a shell-scoping mistake: `NEMOCLAW_INSTALL_TAG=v0.0.26 curl ... | bash` scopes the env var to curl, not to the downstream bash reading from the pipe. The bash side defaulted to `latest`, resolved to v0.0.36, and the install proceeded for several minutes producing convincing-looking output before the mistake surfaced. The install script had no log line saying which version it actually resolved, so the silent failure looked like a working install of v0.0.26. Skill-side fix: after each install (Step 8a baseline and Step 8d latest), echo the resolved `nemoclaw --version` and case-match it against the requested version. Mismatch in baseline sets BASELINE_INSTALL_FAILED=1 to prevent verifying against the wrong version. Mismatch in latest logs a WARN that gets surfaced in the final comment. Cheapest possible guard against the class "I asked for X, got Y" — and the principle generalizes: print the resolved state, never trust the requested state. Also fix the sandbox name in the install.sh-passthrough block: change `NEMOCLAW_SANDBOX_NAME=__verify_stale_install__` (rejected by NemoClaw's name validator: "Allowed format: lowercase, starts with a letter, letters/numbers/internal hyphens only, ends with letter/number") to `NEMOCLAW_SANDBOX_NAME=verify-stale-install`. Surfaced during the #2519 e2e run; net-zero impact on that run because install.sh fell through on name validation rather than reaching the buggy code path, but the spec was wrong. The deeper context for the resolved-version-echo fix: when the rot hypothesis was being tested, an independent agent's logical analysis correctly identified that the public install URL DOES support `NEMOCLAW_INSTALL_TAG=<ref>` and demanded an empirical re-test. The re-test (with the env var on the bash side of the pipe) installed v0.0.26 correctly and STILL hit `Patch 4 (replaceConfigFile EACCES) not applied` — so the cap-at-84 framing held empirically, just not for the original "git-clone is the only path" reason. The full investigation is documented in findings.md Part 6's closing section ("Closing investigation: is baseline-build rot real, or our setup artifact?"). Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 37 ++++++++++++++++--- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 5c25b4356ad..d706707d6da 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -521,14 +521,33 @@ brev exec "$INSTANCE_NAME" " NEMOCLAW_NON_INTERACTIVE=1 \ NEMOCLAW_PROVIDER=${NEMOCLAW_PROVIDER:-ollama} \ NEMOCLAW_MODEL=${NEMOCLAW_MODEL:-nemotron-3-nano:4b} \ - NEMOCLAW_SANDBOX_NAME=__verify_stale_install__ \ + NEMOCLAW_SANDBOX_NAME=verify-stale-install \ ${NVIDIA_API_KEY:+NVIDIA_API_KEY=$NVIDIA_API_KEY} \ bash -c 'curl -fsSL $INSTALL_URL | bash' " || BASELINE_INSTALL_FAILED=1 -brev exec "$INSTANCE_NAME" "nemoclaw --version" -# The bundled onboard creates a sandbox named __verify_stale_install__ that we don't want. -# Destroy it so the reproducer starts from a clean state. +# Verify the resolved install version matches the requested version. This guards against the +# `VAR=val curl ... | bash` shell-scoping footgun where the env var binds to curl, not the +# downstream bash, and the install silently falls through to "latest". Surfaced during a +# rot-debugging investigation where v0.0.36 was silently installed when v0.0.26 was requested +# and several minutes of "convincing" output ran before anyone noticed. Always print the +# resolved state, never trust the requested state. +RESOLVED=$(brev exec "$INSTANCE_NAME" "bash -lc 'nemoclaw --version'" 2>&1 | tail -1) +echo "[verify-stale] baseline requested: $REPORTED_VERSION; resolved: $RESOLVED" +case "$RESOLVED" in + *"$REPORTED_VERSION"*) ;; # match — proceed + *) + echo "ERROR: baseline install resolved to '$RESOLVED' but $REPORTED_VERSION was requested." + echo " Common cause: env-var scoping in the install command. Verify the env vars are on" + echo " the BASH side of the curl|bash pipe, not the curl side. Setting" + echo " BASELINE_INSTALL_FAILED=1 to prevent verifying against the wrong version." + BASELINE_INSTALL_FAILED=1 + ;; +esac + +# The bundled onboard creates a sandbox name we don't want carrying through to the reproducer. +# Use a hyphen-only name (NemoClaw's name validator rejects underscores). Destroy it so the +# reproducer starts from a clean state. brev exec "$INSTANCE_NAME" "sg docker -c 'nemoclaw destroy --all --force 2>/dev/null || true'" ``` @@ -717,7 +736,15 @@ brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcri ```bash brev exec "$INSTANCE_NAME" "$RESET" brev exec "$INSTANCE_NAME" "curl -fsSL $INSTALL_URL | bash" -brev exec "$INSTANCE_NAME" "nemoclaw --version" + +# Same resolved-version check as Step 8a — guard against env-var scoping or default fallthrough +# silently installing the wrong version. The latest install should resolve to $LATEST. +RESOLVED=$(brev exec "$INSTANCE_NAME" "bash -lc 'nemoclaw --version'" 2>&1 | tail -1) +echo "[verify-stale] latest requested: $LATEST; resolved: $RESOLVED" +case "$RESOLVED" in + *"$LATEST"*) ;; # match — proceed + *) echo "WARN: latest install resolved to '$RESOLVED' (expected match for $LATEST). Proceeding but flag in comment." ;; +esac brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./latest-transcript.log From 13bc4c38e534e67d1acb1e345fabac31737c89f1 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 15:58:06 -0700 Subject: [PATCH 26/40] fix(verify-stale): skip issues with active maintainer discussion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a Step 3 skip rule for issues where a MEMBER/OWNER/COLLABORATOR comment was posted within the last 7 days AND the original reporter hasn't replied since. The skill is for *stale* issues; actively- discussed ones are in-flight, and posting a verify-stale verdict on top of an open clarifying question from a maintainer would conflict with their framing and confuse the reporter (now they have two simultaneous asks). Surfaced during pre-flight on issue #2757. The maintainer @cjagwani had just commented questioning the bug's premise — specifically that the reporter's "kill -9 took the parent down" framing didn't line up with how the gateway is launched (`nohup ... &` detaches it) — and asked the reporter to confirm what they actually observed on the Station. Running verify-stale would have posted a "by-design, close as wontfix" verdict on top of an active "wait, did this really happen?" exchange. Wrong move; the skill should detect this state and skip. Implementation reuses the SEVEN_DAYS_AGO cutoff already computed for the marker-TTL check, fetches the reporter via `gh issue view --json author`, then jq-walks the comments array: find the most recent maintainer comment within the cutoff, check whether any reporter comment exists after that timestamp; if maintainer commented recently AND no reporter reply since, skip. Single-issue mode prints the skip reason and exits friendlily so the maintainer knows why the skill bailed. Batch mode just moves on. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index d706707d6da..09f43bfaff9 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -103,6 +103,35 @@ fi Run this check for every candidate that survived the label-based filters above; drop those whose `RECENT_MARKER` is non-empty. +**Active-maintainer-discussion skip.** Drop the issue if a `MEMBER`, `OWNER`, or `COLLABORATOR` comment was posted within the last 7 days AND the original reporter hasn't replied since. The skill is for *stale* issues; actively-discussed ones are in-flight. Posting verify-stale verdicts on top of an open question from a maintainer creates noise and can conflict with the maintainer's framing. Surfaced during pre-flight on #2757 — the maintainer @cjagwani had just asked the reporter clarifying questions about whether the bug premise was even correct, and running verify-stale would have stomped on that conversation. + +```bash +# Reuse the cutoff from the marker-TTL check above ($SEVEN_DAYS_AGO). +REPORTER=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json author --jq .author.login) + +ACTIVE_DISCUSSION=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ + --jq --arg cutoff "$SEVEN_DAYS_AGO" --arg reporter "$REPORTER" ' + (.comments + | map(select( + (.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR") + and .createdAt > $cutoff)) + | sort_by(.createdAt) | last) as $maint + | if $maint == null then null + else + ((.comments + | map(select(.author.login == $reporter and .createdAt > $maint.createdAt)) + | length) as $replies + | if $replies > 0 then null else $maint.createdAt end) + end') + +if [ -n "$ACTIVE_DISCUSSION" ] && [ "$ACTIVE_DISCUSSION" != "null" ]; then + echo "Skip: active maintainer discussion since $ACTIVE_DISCUSSION (reporter has not replied)" + # Single-issue mode: exit 0 with the message; batch mode: continue to next candidate. +fi +``` + +Applies to both single-issue and batch mode. Single-issue mode shows the message and exits friendlily so the maintainer knows why the skill bailed. Batch mode just moves on. + **Candidate rule:** keep the issue if **either**: - The reported version (parsed from body or labels — see Step 4) is **at least 2 versions behind** `$LATEST` in the rightmost-incrementing component, **or** From 22a3997fc3fee51e1588a9561427731cb1e383c9 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 17:06:09 -0700 Subject: [PATCH 27/40] fix(verify-stale): file-based API key propagation, never via cmdline argv MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The post-#2592 commit (1d625dd8) added a Step 5 prompt asking the maintainer to "Export NVIDIA_API_KEY=<key>" and propagate it via `brev exec`. During the #2604 e2e run that played out as `NVIDIA_API_KEY=<value> brev exec ...` — which puts the literal key in the brev exec process's argv, visible in `ps -ef` to anyone with shell access on either the maintainer's laptop or the Brev box for the duration of the run. The skill's stated promise ("never logged") got violated by argv visibility. Fix: switch propagation to file-based. Maintainer writes the key to ~/.nvidia-api-key with 600 perms on their laptop; Step 6.5 brev-copy's the file to the Brev box (encrypted SSH); install / reproducer scripts inside the box read with `NVIDIA_API_KEY=$(cat ~/.nvidia-api-key)`, which sets the env var in the script's own process — never on a command line and never visible in `ps -ef`. Update Step 5's option-1 prompt to instruct the maintainer to use the file form and to `rm ~/.nvidia-api-key` after the run. Add a new "API-key propagation pattern" section after Step 5 that documents the file-based mechanism explicitly. Update Step 8a (baseline install) and Step 8d (latest install) to source the key from the file inside the Brev box's exec context, not from the local shell's env. Note in the docs: if the key was previously propagated via the cmdline (pre-fix), treat it as exposed and rotate. The #2604 run did this; the maintainer was reminded to rotate the NIM key after the run. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 50 +++++++++++++++---- 1 file changed, 40 insertions(+), 10 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 09f43bfaff9..75a9d52b810 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -252,9 +252,18 @@ Set `BUG_PROVIDER=<provider>`. The reporter's reproducer uses the <provider> provider, which requires a real API key to verify faithfully. Three options: - 1. Provide an API key now. Export NVIDIA_API_KEY=<key> (or GEMINI_API_KEY=<key>, etc.) - in the environment running this skill, then re-run. The key is propagated to the - Brev box via `brev exec` and removed when the box is deleted. + 1. Provide an API key via file (NEVER on the command line — keys in argv are + visible in `ps -ef` to anyone with shell access on either machine). Write + the key to a 600-perm file on your laptop: + + printf '%s' '<your-key>' > ~/.nvidia-api-key + chmod 600 ~/.nvidia-api-key + + The skill copies the file to the Brev box via `brev copy` (encrypted SSH), + reads it inside the box with `NVIDIA_API_KEY=$(cat ~/.nvidia-api-key)`, + and never puts the value on a command line. Box deletion removes the file + from the box; you should `rm ~/.nvidia-api-key` on your laptop after the + run. 2. Substitute Ollama and accept the -30 confidence penalty (per Step 8a.5). The verdict will be capped because we're not exercising the real provider's code @@ -268,6 +277,22 @@ Choose 1, 2, or 3: This prompt blocks before Step 7 provisions a box. Don't burn cost on a verification path the maintainer hasn't agreed to. +**API-key propagation pattern (for option 1).** Surfaced during the #2604 e2e run: passing the key as `NVIDIA_API_KEY=<value> brev exec ...` puts the literal value in the brev exec process's argv, which is visible in `ps -ef` on both the maintainer's laptop and the Brev box for the entire duration of the run. That violates the "never logged" promise. The correct pattern is file-based: + +```bash +# After Step 6.5 preconditions, copy the local key file to the Brev box. +[ -f ~/.nvidia-api-key ] && brev copy ~/.nvidia-api-key "$INSTANCE_NAME":~/.nvidia-api-key +brev exec "$INSTANCE_NAME" "chmod 600 ~/.nvidia-api-key 2>/dev/null || true" + +# In setup / reproducer scripts running on the Brev box, source the key from the file. +if [ -f ~/.nvidia-api-key ]; then + export NVIDIA_API_KEY=$(cat ~/.nvidia-api-key) +fi +NEMOCLAW_PROVIDER=build NEMOCLAW_MODEL=<model> nemoclaw onboard ... +``` + +Cleanup: when the trap fires `brev delete`, the box (and the key file on it) goes away. On the maintainer's laptop, the file persists until they `rm ~/.nvidia-api-key` — Step 12's session log should remind them. **If the key was previously propagated via cmdline (pre-fix), treat it as exposed and rotate.** + **Pure-CLI / pure-sandbox-build bugs are exempt** — those don't actually exercise inference, so the provider doesn't matter even if the issue body mentions one. Heuristic: if Step 6.7's local-first predicate would have fired (no sandbox state, no model server interaction), skip the prompt. --- @@ -545,14 +570,16 @@ brev exec "$INSTANCE_NAME" "$RESET" # and either succeeds (ideal) or fails on a real Dockerfile/sandbox-build issue (which # is what we want to detect). Pass NVIDIA_API_KEY only if the maintainer provided one # at Step 5's prompt. +# Read NVIDIA_API_KEY from ~/.nvidia-api-key on the BOX (not from this shell's argv). +# The Step 5 propagation block already brev-copy'd the key file with 600 perms. brev exec "$INSTANCE_NAME" " + if [ -f ~/.nvidia-api-key ]; then export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key); fi NEMOCLAW_INSTALL_TAG=$REPORTED_VERSION \ - NEMOCLAW_NON_INTERACTIVE=1 \ - NEMOCLAW_PROVIDER=${NEMOCLAW_PROVIDER:-ollama} \ - NEMOCLAW_MODEL=${NEMOCLAW_MODEL:-nemotron-3-nano:4b} \ - NEMOCLAW_SANDBOX_NAME=verify-stale-install \ - ${NVIDIA_API_KEY:+NVIDIA_API_KEY=$NVIDIA_API_KEY} \ - bash -c 'curl -fsSL $INSTALL_URL | bash' + NEMOCLAW_NON_INTERACTIVE=1 \ + NEMOCLAW_PROVIDER=${NEMOCLAW_PROVIDER:-ollama} \ + NEMOCLAW_MODEL=${NEMOCLAW_MODEL:-nemotron-3-nano:4b} \ + NEMOCLAW_SANDBOX_NAME=verify-stale-install \ + bash -c 'curl -fsSL $INSTALL_URL | bash' " || BASELINE_INSTALL_FAILED=1 # Verify the resolved install version matches the requested version. This guards against the @@ -764,7 +791,10 @@ brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcri ```bash brev exec "$INSTANCE_NAME" "$RESET" -brev exec "$INSTANCE_NAME" "curl -fsSL $INSTALL_URL | bash" +brev exec "$INSTANCE_NAME" " + if [ -f ~/.nvidia-api-key ]; then export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key); fi + curl -fsSL $INSTALL_URL | bash +" # Same resolved-version check as Step 8a — guard against env-var scoping or default fallthrough # silently installing the wrong version. The latest install should resolve to $LATEST. From fe363cf9716c150109b2f218514593bc952cf90e Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 17:08:57 -0700 Subject: [PATCH 28/40] fix(verify-stale): elevate the comment-authoring principle into Step 10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace Step 10's narrow "Length target" rule with a richer "Comment authoring principle" block that captures lessons accumulated across the e2e runs. The core rule: every section in a rendered comment must either change a reader's mind about the verdict or be cut. Word counts follow from that — 500 is a ceiling, not a target. Most comments land in 200-400. Simple cases land under 200. Document why this matters: comments posted by the skill compete for a maintainer's attention against every other in-flight thread on the repo, and AI-slop prose actively reduces signal-to-noise. Add a worked-examples block citing two real iterations: - #2007 first draft was 750 words; cut to 371. - #2604 took THREE drafts before settling on 190 words because each draft padded with prose that didn't ground the verdict — which caused the verdict itself to drift. Rule learned: name the verdict in one sentence first, then cut any section that doesn't support it. Add per-verdict length targets: - fixed-on-latest: 200-400 words - wontfix (by-design): 250-500 words - verify-inconclusive: 100-200 words - Still-reproduces (no label): 30-80 words — and no transcripts (issue body has them), no @-mention (reporter knows), no architectural prose. One sentence + marker. Add an explicit "cut, by default" list naming the patterns that historically padded comments without changing verdicts. Generalizes the existing memory note into the skill body so future agents reading SKILL.md inherit the principle directly. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 75a9d52b810..7a3ee726aa8 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -1191,7 +1191,30 @@ Transcripts and synth-repro scripts are already plain text and skip the pre-pass **File paths under the reporter's home directory** (`/Users/<name>/`, `/home/<name>/`) → replace with `~/`. Run last; catches incidental username PII. -**Length target.** Default rendered comment to **400–500 words**. The evidence table (or by-design "What's structurally fixed" + "Vestigial references" sections) is the hero. Strip architectural prose "for QA reference," PR-attribution caveats beyond one sentence, and closing reopen-instructions boilerplate. If a comment runs past 500 words, cut everything that doesn't directly support the verdict — every section needs to either change a reader's mind about the verdict or be deleted. +**Comment authoring principle.** Every section in a rendered comment must either change a reader's mind about the verdict, or be cut. Word counts follow from that — 500 is a **ceiling**, not a target. Most fixed-on-latest and by-design comments land in the 200–400 range; simple cases (clear PR ref, deterministic check) land under 200. The principle generalizes: comments posted by this skill compete for a maintainer's attention against every other in-flight thread, and "AI-slop" prose — architectural sidebars, file:line citations the maintainer can find via the PR ref, bare-output reproductions when the load-bearing evidence is elsewhere, "if this verification is wrong, please reopen…" boilerplate — actively reduces the comment's signal-to-noise ratio. + +**For each section in a draft, ask: would the maintainer reach a different conclusion *without* this section? If no, delete.** Lessons accumulated from real runs: + +- **#2007 first draft (~750 words):** had a multi-paragraph "Architectural notes for QA reference" section that didn't change the verdict. Cut → 371 words. +- **#2604 first three drafts:** wavered between fixed-on-latest, still-reproduces, and by-design across iterations because each draft padded the verdict with prose that didn't ground it. Final 190-word draft cut a maintainer-note sidebar about platform attribution, a bare-status output reproduction, and a file:line citation of the source — none affected the verdict, all were AI-slop padding. Rule learned: **before drafting any prose, name the verdict in one sentence; if a section doesn't directly support that one sentence, cut it before writing it.** + +**Per-verdict length defaults:** + +| Verdict | Target | Rationale | +|---|---|---| +| `fixed-on-latest` | 200–400 words | Header + evidence + verdict + @-mention. Add hardware-substitution caveat or related-failure-mode section only if they shift the maintainer's read. | +| `wontfix` (by-design) | 250–500 words | Needs the structurally-fixed + vestigial + what's-not-the-same-bug sections to land cleanly. Skip the bug-report-quote-back if the issue is short. | +| **`verify-inconclusive`** | 100–200 words | One paragraph naming what the skill couldn't establish. No transcripts beyond a single quoted line. | +| **Still-reproduces (no label)** | **30–80 words** | The reporter already has the symptom; the maintainer can see the issue is open. The skill is just confirming + setting the TTL marker. **No transcripts** (the issue body has them), **no @-mention** (the reporter knows their bug is real), **no architectural prose**. One sentence stating "skill ran reproducer on `<latest>`, symptom still present" + one sentence on any partial-fix PR if relevant + marker. That's it. | + +**Cut, by default:** + +- Maintainer-note sidebars about labels / platform attribution unrelated to the bug surface. +- Bare-output reproductions when the load-bearing evidence is in a different command's output. +- File:line citations of source code already findable via the cited PR. +- Closing "if this verification is wrong, please reopen…" boilerplate. +- Redundant verbal framing of what the evidence already shows ("the table above proves…"). +- "Verification mode" pleasantries beyond one factual line. **Mandatory cap caveat.** When the score is capped (Step 9 baseline-validation gating, or any Step 11 degraded-mode path), the rendered Verdict section must include a one-line caveat naming the cap and the reason. Example: `Capped at 84 because Step 9's baseline-validation gate did not run (sandbox-build rot on v0.0.18: Dockerfile symlink layer removed by #2227).` Don't make readers reverse-engineer why the score didn't go higher — name it. From b29958a1d881f8f67894fb76cdcca4e6341add92 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 17:14:37 -0700 Subject: [PATCH 29/40] =?UTF-8?q?fix(verify-stale):=20tighten=20per-verdic?= =?UTF-8?q?t=20length=20targets=20=E2=80=94=20300=20is=20a=20hard=20ceilin?= =?UTF-8?q?g?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Followup to fe363cf9. The 200-400 / 250-500 ranges in the per-verdict table were still too permissive. Tighten to 200-300 for both fixed-on-latest and wontfix; verify-inconclusive stays at 100-200; still-reproduces stays at 30-80. Update the principle preamble: "300 is a hard ceiling for the main verdicts." Simple cases land under 200. If a draft is past 300, it's padding — cut before re-reading. Memory note (feedback_verify_stale_comment_length.md) updated to match. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 7a3ee726aa8..b901e850995 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -1191,7 +1191,7 @@ Transcripts and synth-repro scripts are already plain text and skip the pre-pass **File paths under the reporter's home directory** (`/Users/<name>/`, `/home/<name>/`) → replace with `~/`. Run last; catches incidental username PII. -**Comment authoring principle.** Every section in a rendered comment must either change a reader's mind about the verdict, or be cut. Word counts follow from that — 500 is a **ceiling**, not a target. Most fixed-on-latest and by-design comments land in the 200–400 range; simple cases (clear PR ref, deterministic check) land under 200. The principle generalizes: comments posted by this skill compete for a maintainer's attention against every other in-flight thread, and "AI-slop" prose — architectural sidebars, file:line citations the maintainer can find via the PR ref, bare-output reproductions when the load-bearing evidence is elsewhere, "if this verification is wrong, please reopen…" boilerplate — actively reduces the comment's signal-to-noise ratio. +**Comment authoring principle.** Every section in a rendered comment must either change a reader's mind about the verdict, or be cut. Word counts follow from that — **300 is a hard ceiling** for the main verdicts (fixed-on-latest, wontfix). Simple cases (clear PR ref, deterministic check) land under 200. The principle generalizes: comments posted by this skill compete for a maintainer's attention against every other in-flight thread, and "AI-slop" prose — architectural sidebars, file:line citations the maintainer can find via the PR ref, bare-output reproductions when the load-bearing evidence is elsewhere, "if this verification is wrong, please reopen…" boilerplate — actively reduces the comment's signal-to-noise ratio. **For each section in a draft, ask: would the maintainer reach a different conclusion *without* this section? If no, delete.** Lessons accumulated from real runs: @@ -1202,9 +1202,9 @@ Transcripts and synth-repro scripts are already plain text and skip the pre-pass | Verdict | Target | Rationale | |---|---|---| -| `fixed-on-latest` | 200–400 words | Header + evidence + verdict + @-mention. Add hardware-substitution caveat or related-failure-mode section only if they shift the maintainer's read. | -| `wontfix` (by-design) | 250–500 words | Needs the structurally-fixed + vestigial + what's-not-the-same-bug sections to land cleanly. Skip the bug-report-quote-back if the issue is short. | -| **`verify-inconclusive`** | 100–200 words | One paragraph naming what the skill couldn't establish. No transcripts beyond a single quoted line. | +| `fixed-on-latest` | **200–300 words** | Header + evidence + verdict + @-mention. Add hardware-substitution caveat or related-failure-mode section only if they shift the maintainer's read. If you're past 300, you're padding. | +| `wontfix` (by-design) | **200–300 words** | Structurally-fixed + vestigial + what's-not-the-same-bug, each one to two sentences max. The PR ref carries the detail; the comment carries the verdict. | +| `verify-inconclusive` | 100–200 words | One paragraph naming what the skill couldn't establish. No transcripts beyond a single quoted line. | | **Still-reproduces (no label)** | **30–80 words** | The reporter already has the symptom; the maintainer can see the issue is open. The skill is just confirming + setting the TTL marker. **No transcripts** (the issue body has them), **no @-mention** (the reporter knows their bug is real), **no architectural prose**. One sentence stating "skill ran reproducer on `<latest>`, symptom still present" + one sentence on any partial-fix PR if relevant + marker. That's it. | **Cut, by default:** From f0ec6b8eb60f0847107563097bc7800789d06538 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 17:24:16 -0700 Subject: [PATCH 30/40] fix(verify-stale): two-tier maintainer-discussion handling, unanswered-question variant MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Previously skipped any issue with a maintainer comment in the last 7 days the reporter hadn't replied to. After 7 days the question is no longer "active discussion" — it's a stuck thread, and the skill can be the unsticking voice rather than a clueless interruption. Step 3: classifies the most-recent-unanswered-maintainer-comment as recent (within 7d → skip) or stale (>7d → proceed with variant), exporting UNANSWERED_MAINT_LOGIN/URL/DATE for the templater. Step 10: new mandatory block describing the variant — prepend an "@<maint>'s comment from <date> is still unanswered" lead paragraph and swap the closing reporter-only @-mention for a dual maintainer+reporter @-mention that flags the open question. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 64 +++++++++++++++---- 1 file changed, 52 insertions(+), 12 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index b901e850995..ef44c70518e 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -103,34 +103,62 @@ fi Run this check for every candidate that survived the label-based filters above; drop those whose `RECENT_MARKER` is non-empty. -**Active-maintainer-discussion skip.** Drop the issue if a `MEMBER`, `OWNER`, or `COLLABORATOR` comment was posted within the last 7 days AND the original reporter hasn't replied since. The skill is for *stale* issues; actively-discussed ones are in-flight. Posting verify-stale verdicts on top of an open question from a maintainer creates noise and can conflict with the maintainer's framing. Surfaced during pre-flight on #2757 — the maintainer @cjagwani had just asked the reporter clarifying questions about whether the bug premise was even correct, and running verify-stale would have stomped on that conversation. +**Unanswered-maintainer-question handling.** Find the most recent maintainer (`MEMBER`, `OWNER`, `COLLABORATOR`) comment that the reporter has not replied to since. The age of that comment determines whether the skill skips or proceeds, with a different comment shape if it proceeds: + +- **Within 7 days:** **skip the issue** — the discussion is active, the skill running on top would conflict with the maintainer's framing or confuse the reporter. Surfaced during pre-flight on #2757; running verify-stale on top of a fresh "let me clarify what you observed" question from @cjagwani would have stomped on that conversation. +- **Older than 7 days:** **proceed with verification, but use the unanswered-question comment variant.** After 7 days the maintainer's question has either been forgotten or the reporter has dropped the ball; an independent skill verdict becomes the *unsticking voice* rather than a clueless interruption. The comment leads with "[@<maint>'s question from N days ago](url) is still unanswered" and @-mentions BOTH the maintainer and the reporter, not just the reporter. + +Reuse the `$SEVEN_DAYS_AGO` cutoff from the marker-TTL check above for portability — no cross-platform date math beyond what's already in scope. ```bash -# Reuse the cutoff from the marker-TTL check above ($SEVEN_DAYS_AGO). REPORTER=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json author --jq .author.login) -ACTIVE_DISCUSSION=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ - --jq --arg cutoff "$SEVEN_DAYS_AGO" --arg reporter "$REPORTER" ' +# Most recent unanswered maintainer comment, with age-relative-to-cutoff classification. +UNANSWERED_MAINT=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ + --jq --arg reporter "$REPORTER" --arg cutoff "$SEVEN_DAYS_AGO" ' (.comments - | map(select( - (.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR") - and .createdAt > $cutoff)) + | map(select(.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR")) | sort_by(.createdAt) | last) as $maint | if $maint == null then null else ((.comments | map(select(.author.login == $reporter and .createdAt > $maint.createdAt)) | length) as $replies - | if $replies > 0 then null else $maint.createdAt end) + | if $replies > 0 then null + else { + createdAt: $maint.createdAt, + url: $maint.url, + login: $maint.author.login, + recent: ($maint.createdAt > $cutoff) + } + end) end') -if [ -n "$ACTIVE_DISCUSSION" ] && [ "$ACTIVE_DISCUSSION" != "null" ]; then - echo "Skip: active maintainer discussion since $ACTIVE_DISCUSSION (reporter has not replied)" - # Single-issue mode: exit 0 with the message; batch mode: continue to next candidate. +if [ -n "$UNANSWERED_MAINT" ] && [ "$UNANSWERED_MAINT" != "null" ]; then + MAINT_RECENT=$(printf '%s' "$UNANSWERED_MAINT" | jq -r .recent) + MAINT_DATE=$(printf '%s' "$UNANSWERED_MAINT" | jq -r .createdAt) + MAINT_LOGIN=$(printf '%s' "$UNANSWERED_MAINT" | jq -r .login) + MAINT_URL=$(printf '%s' "$UNANSWERED_MAINT" | jq -r .url) + + if [ "$MAINT_RECENT" = "true" ]; then + echo "Skip: active maintainer discussion (unanswered comment from @$MAINT_LOGIN at $MAINT_DATE, within 7 days)" + # Single-issue mode: exit 0 with the message; batch mode: continue to next candidate. + else + echo "[verify-stale] proceeding with unanswered-question variant — @$MAINT_LOGIN's comment from $MAINT_DATE is older than 7 days" + # Step 10's comment template will lead with the unanswered-question prefix and @-mention + # both the maintainer and the reporter. Export these for the templater: + export UNANSWERED_MAINT_LOGIN="$MAINT_LOGIN" + export UNANSWERED_MAINT_URL="$MAINT_URL" + export UNANSWERED_MAINT_DATE="$MAINT_DATE" + fi fi ``` -Applies to both single-issue and batch mode. Single-issue mode shows the message and exits friendlily so the maintainer knows why the skill bailed. Batch mode just moves on. +When the unanswered-question variant fires (`UNANSWERED_MAINT_LOGIN` set), Step 10's comment template prepends a lead paragraph: + +> [@<maint>'s comment](url) from YYYY-MM-DD is still unanswered. Independent verification below. + +…and the closing @-mention block names BOTH the maintainer (acknowledging their question) and the reporter (asking for confirmation per the standard pattern), instead of just the reporter. **Candidate rule:** keep the issue if **either**: @@ -1230,6 +1258,18 @@ Transcripts and synth-repro scripts are already plain text and skip the pre-pass The skill cannot independently confirm a closed-as-fixed verdict — only the reporter knows whether their original symptom is gone in their environment. The @-mention is what converts a "skill says it's fixed" claim into actionable confirmation work for QA. Customize `<Z>` per case (the version that shipped the fix or `$LATEST`), but never omit the line. +**Mandatory unanswered-question prefix and dual @-mention.** When Step 3 sets `UNANSWERED_MAINT_LOGIN` (a maintainer's question is older than 7 days and the reporter never replied), the verdict comment changes shape in two places: + +1. **Prepend a lead paragraph** as the very first line of the body, before the `## Stale-issue verification` heading: + + > [@\<UNANSWERED_MAINT_LOGIN\>'s comment](\<UNANSWERED_MAINT_URL\>) from \<UNANSWERED_MAINT_DATE\> is still unanswered. Posting independent verification below to unstick the thread. + +2. **Replace the closing reporter-only @-mention with a dual @-mention** that names BOTH the maintainer (acknowledging the open question) and the reporter (per the standard confirmation pattern): + + > @\<UNANSWERED_MAINT_LOGIN\> — flagging that your question above is still open; the verification below may answer it. @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. + +This applies to all three templates (fixed, still-reproduces, by-design). The skill becomes the *unsticking voice* on a thread that has gone quiet — never a clueless interruption when discussion is fresh (Step 3 already filtered the within-7-day case). + **Comment template (fixed / inconclusive — bug not reproduced on latest):** ````markdown From 4ee3750d462c5d400fd786d9b75385f9f613eaa7 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 18:31:37 -0700 Subject: [PATCH 31/40] fix(verify-stale): close on-box subshell argv leak for API keys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Layer 1 (local → Brev) was fixed in 22a3997f via brev copy. Layer 2 (on-box subshell) was unaddressed: scripts running on the Brev box that called sg docker -c "...$NVIDIA_API_KEY..." re-introduced argv exposure because the outer double-quoted heredoc interpolates the key value into the inner shell's argv, visible in ps -ef on the box for the onboard duration. Surfaced during #2611 e2e run when the just-rotated key landed verbatim in `sg docker -c` argv. The right pattern: escape the $ so the inner shell evaluates `cat ~/.nvidia-api-key` itself. The skill section now documents both layers with concrete WRONG / RIGHT side-by-side, and generalizes the rule to any command-string- taking invocation (sg, bash -c, su -c, ssh). Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 33 +++++++++++++++---- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index ef44c70518e..6425f7d87dd 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -305,21 +305,40 @@ Choose 1, 2, or 3: This prompt blocks before Step 7 provisions a box. Don't burn cost on a verification path the maintainer hasn't agreed to. -**API-key propagation pattern (for option 1).** Surfaced during the #2604 e2e run: passing the key as `NVIDIA_API_KEY=<value> brev exec ...` puts the literal value in the brev exec process's argv, which is visible in `ps -ef` on both the maintainer's laptop and the Brev box for the entire duration of the run. That violates the "never logged" promise. The correct pattern is file-based: +**API-key propagation pattern (for option 1).** Argv exposure is a two-layer problem and the file-based pattern must extend to both layers. + +**Layer 1 — local → Brev (surfaced #2604).** Passing the key as `NVIDIA_API_KEY=<value> brev exec ...` puts the literal value in the brev exec process's argv on the maintainer's laptop *and* on the Brev box (since brev exec serializes argv to the remote shell). Visible in `ps -ef` on both ends for the duration of the run. Use file-based copy: ```bash # After Step 6.5 preconditions, copy the local key file to the Brev box. [ -f ~/.nvidia-api-key ] && brev copy ~/.nvidia-api-key "$INSTANCE_NAME":~/.nvidia-api-key brev exec "$INSTANCE_NAME" "chmod 600 ~/.nvidia-api-key 2>/dev/null || true" +``` -# In setup / reproducer scripts running on the Brev box, source the key from the file. -if [ -f ~/.nvidia-api-key ]; then - export NVIDIA_API_KEY=$(cat ~/.nvidia-api-key) -fi -NEMOCLAW_PROVIDER=build NEMOCLAW_MODEL=<model> nemoclaw onboard ... +**Layer 2 — on-box subshell (surfaced #2611).** Inside scripts running on the Brev box, the outer shell reads the key from `~/.nvidia-api-key` cleanly, but a *naive* inner subshell call leaks it back into argv: + +```bash +# WRONG — the double-quoted outer heredoc interpolates $NVIDIA_API_KEY at +# script-eval time, so the literal nvapi- value lands in `sg docker -c "..."`'s +# argv and shows up in `ps -ef` on the box for the whole onboard window. +NVIDIA_API_KEY=$(cat ~/.nvidia-api-key) +sg docker -c " + export NVIDIA_API_KEY='$NVIDIA_API_KEY' # ← argv leak + nemoclaw onboard ... +" + +# RIGHT — escape the $ so the outer shell does not interpolate, and let the +# inner subshell read the file itself. Argv contains the command string +# `cat ~/.nvidia-api-key`, not the value. +sg docker -c " + export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key) + nemoclaw onboard ... +" ``` -Cleanup: when the trap fires `brev delete`, the box (and the key file on it) goes away. On the maintainer's laptop, the file persists until they `rm ~/.nvidia-api-key` — Step 12's session log should remind them. **If the key was previously propagated via cmdline (pre-fix), treat it as exposed and rotate.** +The same rule applies to any `bash -c "..."`, `bash -lc "..."`, `su -c "..."`, `ssh host "..."`, or other invocation that takes a command string as a single argv element: **never interpolate the key into the string at the outer shell's eval time**. Read the file inside the inner shell so the value lives in env-vars, never in argv. + +Cleanup: when the trap fires `brev delete`, the box (and the key file on it) goes away. On the maintainer's laptop, the file persists until they `rm ~/.nvidia-api-key` — Step 12's session log should remind them. **If the key was previously propagated via cmdline (pre-fix at either layer), treat it as exposed and rotate.** **Pure-CLI / pure-sandbox-build bugs are exempt** — those don't actually exercise inference, so the provider doesn't matter even if the issue body mentions one. Heuristic: if Step 6.7's local-first predicate would have fired (no sandbox state, no model server interaction), skip the prompt. From 3abffd72b27e22a3a10c926507dbf997a35ddf0b Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 18:41:27 -0700 Subject: [PATCH 32/40] fix(verify-stale): unbreak markdown link checker on placeholder examples MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three skill paragraphs used `[label](placeholder)` syntax to describe the shape of a markdown link the templater would render. The link checker parses bracket-paren syntax everywhere except fenced code blocks (single backticks aren't enough), so `(<url>)` and `(<UNANSWERED_MAINT_URL>)` were flagged as broken local links. Restructure: fenced code blocks for the literal template line in Step 10, plain prose in Step 3 referencing the Step 10 template instead of duplicating the link shape inline. Verified locally with `bash test/e2e/e2e-cloud-experimental/check-docs.sh --only-links --local-only` — passes cleanly. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 6425f7d87dd..b0cebe2c4f7 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -106,7 +106,7 @@ Run this check for every candidate that survived the label-based filters above; **Unanswered-maintainer-question handling.** Find the most recent maintainer (`MEMBER`, `OWNER`, `COLLABORATOR`) comment that the reporter has not replied to since. The age of that comment determines whether the skill skips or proceeds, with a different comment shape if it proceeds: - **Within 7 days:** **skip the issue** — the discussion is active, the skill running on top would conflict with the maintainer's framing or confuse the reporter. Surfaced during pre-flight on #2757; running verify-stale on top of a fresh "let me clarify what you observed" question from @cjagwani would have stomped on that conversation. -- **Older than 7 days:** **proceed with verification, but use the unanswered-question comment variant.** After 7 days the maintainer's question has either been forgotten or the reporter has dropped the ball; an independent skill verdict becomes the *unsticking voice* rather than a clueless interruption. The comment leads with "[@<maint>'s question from N days ago](url) is still unanswered" and @-mentions BOTH the maintainer and the reporter, not just the reporter. +- **Older than 7 days:** **proceed with verification, but use the unanswered-question comment variant.** After 7 days the maintainer's question has either been forgotten or the reporter has dropped the ball; an independent skill verdict becomes the *unsticking voice* rather than a clueless interruption. The comment leads with a markdown link to the maintainer's unanswered comment (shape shown in the Step 10 template below) and @-mentions BOTH the maintainer and the reporter, not just the reporter. Reuse the `$SEVEN_DAYS_AGO` cutoff from the marker-TTL check above for portability — no cross-platform date math beyond what's already in scope. @@ -154,11 +154,7 @@ if [ -n "$UNANSWERED_MAINT" ] && [ "$UNANSWERED_MAINT" != "null" ]; then fi ``` -When the unanswered-question variant fires (`UNANSWERED_MAINT_LOGIN` set), Step 10's comment template prepends a lead paragraph: - -> [@<maint>'s comment](url) from YYYY-MM-DD is still unanswered. Independent verification below. - -…and the closing @-mention block names BOTH the maintainer (acknowledging their question) and the reporter (asking for confirmation per the standard pattern), instead of just the reporter. +When the unanswered-question variant fires (`UNANSWERED_MAINT_LOGIN` set), Step 10's comment template prepends a lead paragraph (exact shape lives with the templates in Step 10), and the closing @-mention block names BOTH the maintainer (acknowledging their question) and the reporter (asking for confirmation per the standard pattern), instead of just the reporter. **Candidate rule:** keep the issue if **either**: @@ -1279,9 +1275,13 @@ The skill cannot independently confirm a closed-as-fixed verdict — only the re **Mandatory unanswered-question prefix and dual @-mention.** When Step 3 sets `UNANSWERED_MAINT_LOGIN` (a maintainer's question is older than 7 days and the reporter never replied), the verdict comment changes shape in two places: -1. **Prepend a lead paragraph** as the very first line of the body, before the `## Stale-issue verification` heading: +1. **Prepend a lead paragraph** as the very first line of the body, before the `## Stale-issue verification` heading. The lead paragraph is a single line: + + ```text + [@UNANSWERED_MAINT_LOGIN's comment](UNANSWERED_MAINT_URL) from UNANSWERED_MAINT_DATE is still unanswered. Posting independent verification below to unstick the thread. + ``` - > [@\<UNANSWERED_MAINT_LOGIN\>'s comment](\<UNANSWERED_MAINT_URL\>) from \<UNANSWERED_MAINT_DATE\> is still unanswered. Posting independent verification below to unstick the thread. + …with the bracketed variables expanded from the values exported by Step 3. 2. **Replace the closing reporter-only @-mention with a dual @-mention** that names BOTH the maintainer (acknowledging the open question) and the reporter (per the standard confirmation pattern): From e48e5733354e40372cc59c6f6d33ec909be67f05 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Thu, 7 May 2026 18:58:23 -0700 Subject: [PATCH 33/40] fix(cut-release-tag): use xargs -r to skip empty input on label sweep When no open issues carry `fixed-on-latest` or `verify-inconclusive`, the previous pipeline still ran `gh issue edit` once with empty stdin, producing a noisy failure. `xargs -r` skips the invocation entirely when input is empty. CodeRabbit suggestion (PR #3063 review). The skill is hand-authored (not autogenerated from docs/), so the fix lands directly in .agents/skills/. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --- .agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md index 5091b5cc4cb..580e13b4b3f 100644 --- a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md @@ -135,7 +135,7 @@ Strip `fixed-on-latest` and `verify-inconclusive` from all open issues so the ne for label in fixed-on-latest verify-inconclusive; do gh issue list --repo NVIDIA/NemoClaw --state open --label "$label" \ --json number -q '.[].number' \ - | xargs -I{} gh issue edit {} --repo NVIDIA/NemoClaw --remove-label "$label" + | xargs -r -I{} gh issue edit {} --repo NVIDIA/NemoClaw --remove-label "$label" done ``` From b21f9a1bf916d524429a309a867d7d9ea811f698 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Sun, 10 May 2026 20:17:32 -0700 Subject: [PATCH 34/40] fix(verify-stale): consolidate post-review changes into one signed commit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Squashes 8 previously-unsigned commits (e0eab59..c688731) on this branch. Those commits were authored from a clone with stale local git config (user.email=test@example.com, commit.gpgsign=false), so they landed on the PR unverified and with an incorrect author. Net file diff is unchanged; only the 3 skill files actually touched by the squashed commits are included here — the two upstream-main merges from the unsigned period are dropped (re-merge cleanly if needed). Squashed commits, in order: - e0eab59 refactor(verify-stale): split SKILL.md to meet 500-line ceiling - 3cb23b0 feat(verify-stale): move fixed-on-latest issues to "Needs Review" - 9b434bd fix(cut-release-tag): conditional label sweep instead of blanket clear - fcf5b6d fix(cut-release-tag): bump stale-threshold to 14 days - d33983a fix(verify-stale): address coderabbit review findings on PR #3063 - 87a348a fix(verify-stale): two skill changes surfaced from #1642 e2e run - e9e347c fix(verify-stale): re-check state == OPEN before posting - c688731 fix(verify-stale): treat all enhancement: prefixed labels as skip Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../SKILL.md | 88 +- .../nemoclaw-maintainer-verify-stale/SKILL.md | 1046 +-------------- .../reference/execution-and-comment.md | 1167 +++++++++++++++++ 3 files changed, 1269 insertions(+), 1032 deletions(-) create mode 100644 .agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md diff --git a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md index 580e13b4b3f..eba035fea7c 100644 --- a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md @@ -127,19 +127,95 @@ git ls-remote --tags origin | grep -E '(<new-version>|latest)' Confirm both tags point to the same commit on the remote. -## Step 7: Sweep Stale-Issue Verification Labels +## Step 7: Conditionally Sweep Stale-Issue Verification Labels -Strip `fixed-on-latest` and `verify-inconclusive` from all open issues so the next `nemoclaw-maintainer-verify-stale` run re-evaluates against the new release. Without this sweep, "latest" drifts and verifications go silently stale. The skill's by-design path uses the existing repo `status: wont-fix` label, which is **not** swept here — that label is also applied for non-skill reasons (scope, priority, dup decisions), so clearing it would erase human triage work. +Strip `fixed-on-latest` from open issues only when the verification has actually gone stale or a regression risk appeared since we verified — never blanket-sweep. A blanket sweep on every release re-verifies labels that were freshly applied yesterday, wasting Brev cost and creating noise. The skill's by-design path uses the existing repo `status: wont-fix` label, which is **not** swept (also applied for non-skill triage reasons, so clearing it would erase human work). `verify-inconclusive` is also kept on the same conditional cascade as `fixed-on-latest`. + +**Decision cascade per labeled-and-open issue:** + +| Order | Check | Action | +|---|---|---| +| 1 | Project [NVIDIA/199](https://github.com/orgs/NVIDIA/projects/199) status == **Done** | **Skip clear** — maintainer already accepted the verification; label can stay until the issue closes. | +| 2 | More than 14 days since the skill marker comment AND status != Done | **Clear** — verification is stale; reporter never confirmed in the review window. Re-verify on next skill run. | +| 3 | A PR merged since the marker date touches the paths the comment cited in `Relevant changes since v0.0.X` | **Clear** — regression risk; what was "fixed" may have been re-broken. | +| — | else | **Skip clear** — verification still holds; skill won't re-run on this issue (still excluded by Step 3 marker-TTL plus the live label). | + +Closed issues are not iterated (the `--state open` filter on the listing excludes them implicitly). + +Requires the `project` scope on the maintainer's gh CLI for the Project 199 status lookup. If missing, run `gh auth refresh -h github.com -s project` in a real terminal once (OAuth device-code flow). With the scope absent, the sweep falls back to the **time + regression** logic alone (skips check #1) and logs a warning. ```bash +PROJECT_NUMBER=199 +TODAY_TS=$(date -u +%s) +HAVE_PROJECT_SCOPE=0 +gh auth status 2>&1 | grep -q "'project'" && HAVE_PROJECT_SCOPE=1 || \ + echo "[release-sweep] WARN gh missing 'project' scope — Done-state check disabled this run" + for label in fixed-on-latest verify-inconclusive; do - gh issue list --repo NVIDIA/NemoClaw --state open --label "$label" \ - --json number -q '.[].number' \ - | xargs -r -I{} gh issue edit {} --repo NVIDIA/NemoClaw --remove-label "$label" + for n in $(gh issue list --repo NVIDIA/NemoClaw --state open --label "$label" --json number -q '.[].number'); do + + # 1. Project Done-state check (only if we have project scope) + if [ "$HAVE_PROJECT_SCOPE" = "1" ]; then + STATUS=$(gh api graphql -F num="$n" -f query=' + query($num: Int!) { + repository(owner: "NVIDIA", name: "NemoClaw") { + issue(number: $num) { + projectItems(first: 10) { + nodes { + project { number } + fieldValueByName(name: "Status") { + ... on ProjectV2ItemFieldSingleSelectValue { name } + } + } + } + } + } + }' --jq '.data.repository.issue.projectItems.nodes[] | select(.project.number == 199) | .fieldValueByName.name' 2>/dev/null | head -1) + if [ "$STATUS" = "Done" ]; then + echo "[release-sweep] kept #$n ($label) — Project 199 status is Done" + continue + fi + fi + + # 2. Find the most recent skill marker comment date + MARKER_DATE=$(gh issue view "$n" --repo NVIDIA/NemoClaw --json comments \ + --jq '.comments | map(select(.body | test("nemoclaw-verify-stale v\\d+ \\d{4}-\\d{2}-\\d{2}"))) | last | .body | (capture("nemoclaw-verify-stale v\\d+ (?<d>\\d{4}-\\d{2}-\\d{2})") // {}) | .d // empty') + if [ -z "$MARKER_DATE" ]; then + # Label exists but no skill marker — applied manually; leave alone. + echo "[release-sweep] kept #$n ($label) — no skill marker, label applied manually" + continue + fi + + AGE_DAYS=$(( (TODAY_TS - $(date -u -j -f "%Y-%m-%d" "$MARKER_DATE" +%s 2>/dev/null || date -u -d "$MARKER_DATE" +%s)) / 86400 )) + if [ "$AGE_DAYS" -ge 14 ]; then + gh issue edit "$n" --repo NVIDIA/NemoClaw --remove-label "$label" + echo "[release-sweep] cleared #$n ($label) — stale (verified ${AGE_DAYS}d ago, reporter not confirmed)" + continue + fi + + # 3. Regression check — any PR-merge commit since MARKER_DATE touch the paths the + # comment's `Relevant changes since v0.0.X` block cited? + PATHS=$(gh issue view "$n" --repo NVIDIA/NemoClaw --json comments \ + --jq '.comments | map(select(.body | test("nemoclaw-verify-stale v\\d+"))) | last | .body' \ + | grep -oE '`[a-zA-Z0-9_/.-]+\.(ts|js|sh|py|yaml|yml|md)`' | tr -d '`' | sort -u) + if [ -n "$PATHS" ]; then + # Run from the current directory — Step 1's prerequisite already requires the maintainer + # to be inside the NemoClaw repo, and hardcoding ~/NemoClaw breaks anyone with a non-default + # checkout location. + REGRESSED=$(git log --since="$MARKER_DATE" origin/main --name-only --format=oneline -- $PATHS 2>/dev/null | head -1) + if [ -n "$REGRESSED" ]; then + gh issue edit "$n" --repo NVIDIA/NemoClaw --remove-label "$label" + echo "[release-sweep] cleared #$n ($label) — regression risk (commits since ${MARKER_DATE} touch implicated paths)" + continue + fi + fi + + echo "[release-sweep] kept #$n ($label) — verified ${AGE_DAYS}d ago, no Done state, no regression touch" + done done ``` -The verification record itself stays in each issue's comment history — only the labels are reset. +The verification record itself stays in each issue's comment history — only the labels are reset, and only when the cascade above fires. ## Important Notes diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index b0cebe2c4f7..0f043abcb9d 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -1,6 +1,6 @@ --- name: nemoclaw-maintainer-verify-stale -description: Verify whether old NVIDIA/NemoClaw bug reports still reproduce against the latest tag. Picks candidate issues opened against older versions, runs the reproducer locally first when possible (Linux or macOS), otherwise reuses or provisions a Brev Linux box (CPU or GPU), detects behavior that was intentionally changed, scores confidence, and posts an evidence-backed comment with a label (fixed-on-latest, status wont-fix, or verify-inconclusive). Tag-only — never auto-closes. Brev verification is Linux-only in v1; Windows and integration-token-dependent issues are skipped. Trigger keywords - verify stale, verify fixed, reproduce on latest, stale issue, old bug, fixed-on-latest, status wont-fix, verify-inconclusive, drain backlog, brev verify. +description: Verify whether old NVIDIA/NemoClaw bug reports still reproduce against the latest tag. Picks candidate issues opened against older versions, runs the reproducer locally first when possible (Linux or macOS), otherwise reuses or provisions a Brev Linux box (CPU or GPU), detects behavior that was intentionally changed, scores confidence, and posts an evidence-backed comment with a label (fixed-on-latest, status: wont-fix, or verify-inconclusive). Tag-only — never auto-closes. Brev verification is Linux-only in v1; Windows and integration-token-dependent issues are skipped. Trigger keywords - verify stale, verify fixed, reproduce on latest, stale issue, old bug, fixed-on-latest, status: wont-fix, verify-inconclusive, drain backlog, brev verify. user_invocable: true --- @@ -63,7 +63,7 @@ This is the version the skill will verify against. Record it — every comment m Apply these rules in order. Drop any issue that fails a rule. **Issue-type allowlist:** must have `bug` label. -**Issue-type skip:** drop if any of `enhancement`, `documentation`, `status: wont-fix`, `status: needs-info`, `security`. Use the canonical repo label names — bare `wontfix` / `needs-info` are NOT the repo's labels (verified via `gh label list`); the actual labels carry a `status:` prefix and a hyphen. +**Issue-type skip:** drop if any label exactly matches `documentation`, `status: wont-fix`, `status: needs-info`, `security`, OR is `enhancement` / starts with the prefix `enhancement:` (the repo has 8 prefixed variants — `enhancement: feature`, `enhancement: MCP`, `enhancement: testing`, `enhancement: ui`, `enhancement: provider`, `enhancement: platform`, `enhancement: policy`, `enhancement: inference`, `enhancement: integration`, `enhancement: performance`, `enhancement: skill` — and exact-match misses them all; surfaced from #1752). Use the canonical repo label names — bare `wontfix` / `needs-info` are NOT the repo's labels (verified via `gh label list`); the actual labels carry a `status:` prefix and a hyphen. **Platform skip (Brev-reproducible only in v1):** drop if any of `Platform: Windows/WSL`, `Platform: MacOS`, `Platform: macOS`, `Platform: Jetson AGX Thor/Orin`. Brev has no equivalent hardware for Jetson (embedded/edge ARM with integrated GPU is not in the Brev SKU catalog), so any Brev verification of a Jetson-only bug would produce a misleading "fixed-on-x86" verdict. Keep `Platform: Ubuntu`, `Platform: DGX Spark`, `Platform: GB10`, `Platform: All`, or no platform label. `Platform: DGX Spark` and `Platform: GB10` stay in scope but Step 10 requires a "Hardware substitution" caveat in the comment naming the Brev SKU we used as a substitute (Brev x86 GPU SKUs are not faithful to GB10 / Grace Hopper silicon for performance-shape or memory-architecture-shape bugs). @@ -103,21 +103,20 @@ fi Run this check for every candidate that survived the label-based filters above; drop those whose `RECENT_MARKER` is non-empty. -**Unanswered-maintainer-question handling.** Find the most recent maintainer (`MEMBER`, `OWNER`, `COLLABORATOR`) comment that the reporter has not replied to since. The age of that comment determines whether the skill skips or proceeds, with a different comment shape if it proceeds: +**Unanswered-maintainer-question handling.** Find the most recent maintainer (`MEMBER`, `OWNER`, `COLLABORATOR`) comment that **looks like a question** (`?`, polite imperative like "please confirm/share/clarify", or starter like "could you / can you / do you") AND that the reporter has not replied to since. Pure triage acknowledgments (`"✨ Thanks for reporting…"`) are skipped. The age of the qualifying comment determines skip-or-proceed: - **Within 7 days:** **skip the issue** — the discussion is active, the skill running on top would conflict with the maintainer's framing or confuse the reporter. Surfaced during pre-flight on #2757; running verify-stale on top of a fresh "let me clarify what you observed" question from @cjagwani would have stomped on that conversation. -- **Older than 7 days:** **proceed with verification, but use the unanswered-question comment variant.** After 7 days the maintainer's question has either been forgotten or the reporter has dropped the ball; an independent skill verdict becomes the *unsticking voice* rather than a clueless interruption. The comment leads with a markdown link to the maintainer's unanswered comment (shape shown in the Step 10 template below) and @-mentions BOTH the maintainer and the reporter, not just the reporter. - -Reuse the `$SEVEN_DAYS_AGO` cutoff from the marker-TTL check above for portability — no cross-platform date math beyond what's already in scope. +- **Older than 7 days:** **proceed with verification, but use the unanswered-question comment variant.** After 7 days the maintainer's question has either been forgotten or the reporter has dropped the ball; an independent skill verdict becomes the *unsticking voice* rather than a clueless interruption. The comment leads with a markdown link to the maintainer's unanswered comment (shape shown in the Step 10 template below) and @-mentions BOTH the maintainer and the reporter, not just the reporter. Reuse `$SEVEN_DAYS_AGO` from the marker-TTL check above. ```bash REPORTER=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json author --jq .author.login) -# Most recent unanswered maintainer comment, with age-relative-to-cutoff classification. +# Most recent unanswered maintainer comment that looks like a question — filters out triage acknowledgments (#1642 surfaced this). UNANSWERED_MAINT=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ --jq --arg reporter "$REPORTER" --arg cutoff "$SEVEN_DAYS_AGO" ' (.comments - | map(select(.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR")) + | map(select((.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR") + and (.body | test("\\?|(?i)\\bplease (confirm|share|provide|clarify|tell|verify|check|let me know|let us know)|(?i)\\b(could|can|would) you\\b|(?i)\\bdo you (have|know|see|use)\\b")))) | sort_by(.createdAt) | last) as $maint | if $maint == null then null else @@ -396,6 +395,9 @@ if [ -z "$GH_IDENTITY" ]; then fi echo "gh identity: @$GH_IDENTITY — comments posted by this run will appear under this handle" +# gh 'project' scope — Step 10 moves fixed-on-latest issues to "Needs Review" on Project 199. Warn if missing. +gh auth status 2>&1 | grep -q "'project'" || echo "[verify-stale] WARN gh missing 'project' scope — Step 10 tracker move will skip. Fix: run 'gh auth refresh -h github.com -s project' in a real terminal." + # Brev auth — short-circuit only after the auth check, not before. # When auth fails, give the user a directive recipe (the browser-flow path is # what works from non-TTY harnesses like Claude Code, not the headless options). @@ -483,1023 +485,15 @@ Compare local result to the issue's "Actual Result" section using the same match --- -## Step 7: Reuse or Provision a Brev Box - -The skill prefers reuse over provisioning. A pool of `verify-stale-*` boxes (CPU and GPU) can be kept warm; reuse the matching one if available, otherwise provision. - -```bash -# Auth + install URL already verified by Step 6.5 — no need to re-check or auto-login here. - -# Determine class from Step 5: "cpu" or "gpu" -INSTANCE_CLASS="cpu" # or "gpu" - -INSTANCES=$(brev ls --json) - -# Look for an existing running verify-stale-* box matching the required class. -# CPU boxes have no .gpu field set; GPU boxes do. -EXISTING=$(echo "$INSTANCES" | jq -r --arg class "$INSTANCE_CLASS" ' - .[]? - | select(.name | startswith("verify-stale-")) - | select(.status == "RUNNING") - | select(($class == "gpu" and (.gpu // "" != "")) - or ($class == "cpu" and (.gpu // "" == ""))) - | .name' | head -1) - -PROVISIONED_NEW=0 - -if [ -n "$EXISTING" ]; then - INSTANCE_NAME="$EXISTING" - echo "Reusing existing verification box: $INSTANCE_NAME" -else - # Concurrency cap: refuse if 4+ verify-stale-* boxes are already running. - RUNNING=$(echo "$INSTANCES" | jq '[.[]? | select(.name | startswith("verify-stale-"))] | length') - if [ "$RUNNING" -ge 4 ]; then - echo "ERROR: 4 verify-stale boxes already running. Wait for one to finish or reuse." - exit 1 - fi - - INSTANCE_NAME="verify-stale-${ISSUE_NUMBER}-$(date +%s)" - - if [ "$INSTANCE_CLASS" = "gpu" ]; then - # brev create auto-selects the cheapest GPU meeting the defaults - # (>=20GB VRAM, >=500GB disk, compute >=8.0). Override with --type if needed. - brev create "$INSTANCE_NAME" - else - # CPU case: pick the cheapest stoppable Linux SKU at runtime so the skill doesn't rot when - # SKUs change. Bias the floor by reproducer-implied memory needs — the cheapest 2 GB SKU - # cannot load a 4.8 GiB Ollama probe, and onboard fails at provider validation before any - # sandbox-creation code runs. Surfaced during the #2007 e2e run (wasted ~25 min on a 2 GB - # box that couldn't load `nemotron-3-nano:4b`). - # - # Memory floor heuristic: - # - Reproducer references Ollama or vLLM or names a model tag (e.g. `nemotron-3-nano:4b`, - # `llama3:8b`) -> floor 16 GB (covers ~5 GB model + sandbox + gateway overhead). - # - Reproducer touches sandbox onboarding without a local model server -> floor 8 GB. - # - Pure CLI-surface bug (no sandbox, no model) -> floor 4 GB. - # Override the auto-pick by exporting VERIFY_STALE_CPU_TYPE if the team has hard preferences. - CPU_RAM_FLOOR=${CPU_RAM_FLOOR:-8} - CPU_TYPE=${VERIFY_STALE_CPU_TYPE:-$(brev search cpu --sort price --json \ - | jq -r --argjson floor "$CPU_RAM_FLOOR" \ - '[.[] | select(.stoppable == true and .ram_gb >= $floor)] | .[0].type')} - [ -n "$CPU_TYPE" ] || { echo "ERROR: no stoppable CPU SKU with >= ${CPU_RAM_FLOOR} GB RAM"; exit 1; } - brev create "$INSTANCE_NAME" --type "$CPU_TYPE" - fi - - PROVISIONED_NEW=1 -fi - -# Cleanup runs on success, error, and SIGINT. -# Delete only what we provisioned. Reused boxes stay warm for next time. -# `brev delete` is non-interactive by default — there is no --yes flag, and passing one errors. -echo ">>> Brev instance: $INSTANCE_NAME (provisioned_new=$PROVISIONED_NEW; manual cleanup: brev delete $INSTANCE_NAME)" -trap '[ "$PROVISIONED_NEW" = "1" ] && brev delete "$INSTANCE_NAME" >/dev/null 2>&1 || true' EXIT -``` - -Wallclock cap per verification: **60 minutes** default. The cap accommodates two full install passes (baseline + latest), comprehensive resets between them, and any reproducer dependency bootstrapping (Step 8a.5) — most of which run sequentially against a single Brev box. Bugs that genuinely require more than an hour to manifest fall out of v1 scope; if a provisioned box isn't ready in time, abort and treat as an infra failure (Step 11). - -The previous design had a 25-min default with a 60-min extension for time-sensitive bugs (`memory leak`, `over time`, etc.). That split optimised for the wrong constraint — most issues fit comfortably under 60 min, and the keyword-based extension forced re-runs whenever a real install or bootstrap took longer than the optimistic 25-min budget. Single 60-min cap removes that paper cut. - ---- - -## Step 8: Validate on Baseline, Verify on Latest - -Two-pass design. - -- **Baseline pass (8a–8c):** install the **reported version**, run the reproducer, confirm it actually exposes the bug as described. This is the gate that proves the script is real. -- **Latest pass (8d):** install **latest**, run the validated reproducer. This is what the confidence score is built on. - -Without the baseline gate, a clean run on latest is ambiguous: maybe the bug really got fixed, maybe the script was never capable of triggering it. The baseline disambiguates. - -### Comprehensive reset (run before each install) - -NemoClaw spawns OpenShell sandboxes (containers), runtime services, and listening processes. A naive `rm -rf ~/.nemoclaw` doesn't clean those — the latest install would inherit baseline state and contaminate the result. Use this fuller reset between installs: - -```bash -RESET=$(cat <<'SCRIPT' -nemoclaw destroy --all --force 2>/dev/null || true -# Anchor pkill patterns to "/nemoclaw" / "/openshell" path components so the kill doesn't -# match unrelated processes that happen to mention these strings (including the agent -# harness running this skill if its working dir contains the word). -pkill -9 -f '/nemoclaw([[:space:]]|$)' 2>/dev/null || true -pkill -9 -f '/openshell([[:space:]]|$)' 2>/dev/null || true -docker ps -a --filter "name=openshell-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true -docker ps -a --filter "name=nemoclaw-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true -# Sandbox state lives in ~/.openclaw (default-writable since #2227); ~/.nemoclaw holds CLI state. -# Wipe both so the latest install starts clean. -rm -rf ~/.nemoclaw ~/.openclaw 2>/dev/null -sudo -n rm -f /usr/local/bin/nemoclaw 2>/dev/null || true -sudo -n rm -rf /usr/local/lib/nemoclaw 2>/dev/null || true -for port in 8080 18789 9119; do fuser -k -n tcp $port 2>/dev/null || true; done -true -SCRIPT -) -``` - -Idempotent — fails silently when there's nothing to clean. Run via `brev exec "$INSTANCE_NAME" "$RESET"` before 8a's install and again before 8d's install. - -**Sudo precondition.** All `sudo` invocations use `sudo -n` (non-interactive) so they fail fast instead of hanging on a password prompt. The skill assumes the Brev image's default user has passwordless sudo configured — Brev's stock images do; custom images may not. If `sudo -n` fails, the binary cleanup is best-effort and a stale `/usr/local/bin/nemoclaw` may persist. The user-local install path (`~/.nemoclaw`) is fully reset regardless. - -### Step 8a: Install reported version - -The installer accepts the target ref via the `NEMOCLAW_INSTALL_TAG` env var (verified against `install.sh` source — defaults to `latest` if unset). It is **not** a `--version` flag. - -```bash -brev exec "$INSTANCE_NAME" "$RESET" - -# Pass the provider env vars through so install.sh's bundled `[3/3] Onboarding` step -# doesn't fall back to the default `build` (NIM) provider — which requires NVIDIA_API_KEY -# and otherwise fails the install with a misleading error. When NEMOCLAW_PROVIDER=ollama -# (the common case), the bundled onboard uses the local Ollama we set up in Step 8a.5 -# and either succeeds (ideal) or fails on a real Dockerfile/sandbox-build issue (which -# is what we want to detect). Pass NVIDIA_API_KEY only if the maintainer provided one -# at Step 5's prompt. -# Read NVIDIA_API_KEY from ~/.nvidia-api-key on the BOX (not from this shell's argv). -# The Step 5 propagation block already brev-copy'd the key file with 600 perms. -brev exec "$INSTANCE_NAME" " - if [ -f ~/.nvidia-api-key ]; then export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key); fi - NEMOCLAW_INSTALL_TAG=$REPORTED_VERSION \ - NEMOCLAW_NON_INTERACTIVE=1 \ - NEMOCLAW_PROVIDER=${NEMOCLAW_PROVIDER:-ollama} \ - NEMOCLAW_MODEL=${NEMOCLAW_MODEL:-nemotron-3-nano:4b} \ - NEMOCLAW_SANDBOX_NAME=verify-stale-install \ - bash -c 'curl -fsSL $INSTALL_URL | bash' -" || BASELINE_INSTALL_FAILED=1 - -# Verify the resolved install version matches the requested version. This guards against the -# `VAR=val curl ... | bash` shell-scoping footgun where the env var binds to curl, not the -# downstream bash, and the install silently falls through to "latest". Surfaced during a -# rot-debugging investigation where v0.0.36 was silently installed when v0.0.26 was requested -# and several minutes of "convincing" output ran before anyone noticed. Always print the -# resolved state, never trust the requested state. -RESOLVED=$(brev exec "$INSTANCE_NAME" "bash -lc 'nemoclaw --version'" 2>&1 | tail -1) -echo "[verify-stale] baseline requested: $REPORTED_VERSION; resolved: $RESOLVED" -case "$RESOLVED" in - *"$REPORTED_VERSION"*) ;; # match — proceed - *) - echo "ERROR: baseline install resolved to '$RESOLVED' but $REPORTED_VERSION was requested." - echo " Common cause: env-var scoping in the install command. Verify the env vars are on" - echo " the BASH side of the curl|bash pipe, not the curl side. Setting" - echo " BASELINE_INSTALL_FAILED=1 to prevent verifying against the wrong version." - BASELINE_INSTALL_FAILED=1 - ;; -esac - -# The bundled onboard creates a sandbox name we don't want carrying through to the reproducer. -# Use a hyphen-only name (NemoClaw's name validator rejects underscores). Destroy it so the -# reproducer starts from a clean state. -brev exec "$INSTANCE_NAME" "sg docker -c 'nemoclaw destroy --all --force 2>/dev/null || true'" -``` - -If install fails (old releases rot — installer URLs, deps, OS images all drift over time, or the in-image Dockerfile patch step asserts against a code shape that's since changed), set `BASELINE_INSTALL_FAILED=1` and **skip 8b/8c**, going straight to 8d. Note "baseline-install-skipped" or "baseline-build-skipped" in the final comment depending on which phase rotted. Step 9's scoring rule handles the degraded mode (cap at 84). - -**The reproducer's own `nemoclaw onboard` (Step 8b) must pass `--fresh`.** If install.sh's bundled onboard was in an in-progress or failed state when we destroyed the install sandbox, the reproducer's onboard would error with `Previous onboarding session failed. Re-run with --fresh to discard it`. `--fresh` ensures a clean start. - -### Step 8a.5: Bootstrap reproducer dependencies - -Brev's stock CPU images ship with NemoClaw installable but not the broader ecosystem the reproducer may need — local model servers (Ollama, vLLM), inference providers, third-party CLIs. **Default to maximum faithfulness: install the actual dependency the reporter used rather than substituting a stub.** Substituting trades faithfulness for speed; that trade is rarely worth it on a 60-min budget, and it almost always introduces a confound that makes the verdict less trustworthy. - -**When to bootstrap (not substitute):** - -- The reproducer references a specific model/server runtime (`NEMOCLAW_PROVIDER=ollama`, `NEMOCLAW_PROVIDER=vllm`, etc.). -- The reproducer references a specific model name with a tag (`nemotron-3-nano:4b`, `llama3:8b`, etc.). -- The reporter's environment in the issue body shows a configured provider (e.g., `OpenShell CLI: 0.0.26` plus an Ollama running on host). - -**When to substitute (with -30 penalty):** - -- Provider requires an API key the skill cannot safely supply (NIM, OpenAI, Anthropic, etc.). Stubbing a key won't pass validation faithfully and a real key shouldn't sit in a verify-stale run. Apply the -30 penalty (treat as synth-repro per Step 8b) and document the substitution in the comment. -- The bug is *provably* independent of the dependency (e.g., a CLI argument-parsing bug that errors before any provider runs). Note this explicitly in the comment. - -**Canonical bootstraps:** - -```bash -# Ollama + a specific model. -# The Ollama installer registers a systemd service (`ollama.service`) so the -# daemon survives between brev exec calls. -brev exec "$INSTANCE_NAME" "curl -fsSL https://ollama.com/install.sh | sh" -brev exec "$INSTANCE_NAME" "sudo systemctl start ollama && sleep 3" -brev exec "$INSTANCE_NAME" "ollama pull <model>" -brev exec "$INSTANCE_NAME" "ollama list" # confirm before continuing -``` - -```bash -# vLLM + a model (HuggingFace-hosted). -brev exec "$INSTANCE_NAME" "pip install --quiet vllm" -brev exec "$INSTANCE_NAME" "nohup python -m vllm.entrypoints.openai.api_server --model <model> --host 127.0.0.1 --port 8000 >/var/log/vllm.log 2>&1 &" -brev exec "$INSTANCE_NAME" "sleep 30 && curl -fsS http://127.0.0.1:8000/v1/models" -``` - -Bootstrap **once before Step 8b's baseline run** and reuse for Step 8d's latest run. Don't reset Ollama/vLLM state between baseline and latest in the comprehensive reset — model downloads are expensive and unrelated to the NemoClaw install. Adjust the reset script to skip these external services explicitly if needed. - -**If bootstrap fails** (network issue pulling the model, service won't start, etc.), this is an infra failure — abort to Step 11. Do not silently substitute; the user opted into faithfulness for a reason. - -**Ollama coverage table.** Ollama is the default provider for verification runs because it's free, local, and self-hosted. It covers most bug classes faithfully but not all. Use this table to decide whether Ollama is sufficient or whether Step 5's API-key prompt should fire: - -| Bug class | Ollama covers? | Notes | -|---|---|---| -| CLI surface (subcommand parsing, flag handling, oclif dispatch) | ✓ Always | Provider not exercised | -| Sandbox structure (build, file permissions, mounts, layout) | ✓ Always | Provider not exercised | -| Networking / policy (port forwards, NAT, egress rules, channels guards) | ✓ Always | Provider not exercised | -| Generic inference flow (does an agent turn complete, does the proxy route correctly) | ✓ Usually | Ollama can fail in the same shape as NIM/Gemini for most flow bugs | -| Provider-specific behavior (`Provider: NVIDIA` symptom, NIM-only error handling, `Provider: Gemini` quirks) | ✗ No | Different code paths; substitution doesn't exercise the bug | -| Model-specific behavior (`gemini-flash-3-preview` doesn't handle prompt X, `nemotron-3-nano:4b` works fine) | ✗ No | Wrong model = wrong outputs | -| Ollama-shape-specific (#2519 "Ollama-local 401" — local-vs-networked Ollama config) | △ Sometimes | A generic Ollama install may or may not reproduce; may need specific configuration | -| Performance / latency on specific silicon | ✗ No | Hardware substitution caveat (Step 10) and Step 8e perf rubric apply | -| Quota / rate-limit / API-key validation | ✗ No | Ollama doesn't have those failure modes | - -When the table says ✗ No or △ Sometimes, Step 5's API-key prompt fires. When it says ✓, proceed with Ollama and skip the prompt. - -### Step 8a.5b: Brev exec environment quirks - -Two non-obvious gotchas surfaced during the #2007 e2e run that every subsequent `brev exec` call has to handle. Encode them once here so reproducer scripts don't have to relearn each time. - -**PATH does not include `~/.local/bin` in non-login shells.** `nemoclaw`'s installer drops a shim at `~/.local/bin/nemoclaw` and updates PATH via `~/.bashrc` / `~/.profile`. `brev exec` spawns non-login, non-interactive shells that don't source those files, so a bare `brev exec "$INSTANCE" "nemoclaw --version"` returns `command not found` on a freshly-installed box. Fix: every reproducer script must explicitly export PATH at the top, OR every `brev exec` call must wrap with `bash -lc '...'`. - -```bash -# Reproducer scripts: prepend this line. -export PATH="$HOME/.local/bin:$PATH" - -# Or equivalently when calling brev exec ad-hoc: -brev exec "$INSTANCE" "bash -lc 'nemoclaw --version'" -``` - -**Docker group requires `sg docker -c '...'` after `usermod -aG`.** Adding the user to the `docker` group (`sudo usermod -aG docker ubuntu`) takes effect for new login sessions, but `brev exec` calls in the same Brev session keep the old gid. The reproducer's `nemoclaw onboard` will fail with `permission denied while connecting to /var/run/docker.sock` unless the call runs in a subshell with the docker group active. - -```bash -# Reproducer execution: wrap with sg docker. -brev exec "$INSTANCE" "sg docker -c 'bash ~/reproducer.sh'" -``` - -Both patterns appear in the canonical setup script committed alongside the skill (or are encoded in your reproducer wrapper). Don't rely on the user discovering them mid-run. - -**`openshell sandbox exec` argument-order footgun.** When the reproducer needs to run a command *inside* the sandbox (channels-guard checks, in-sandbox file inspection, etc.), the correct non-interactive form uses `-n <name>` and a `--` separator: - -```bash -# Correct: -openshell sandbox exec -n ai -- bash -c 'source /sandbox/.bashrc; openclaw channels add telegram; echo "EXIT=$?"' - -# Wrong (silently auto-detects sandbox by "last used", stuffs the leftover positional -# `ai` into bash's $0, prints "/bin/bash: line 1: ai: command not found" — the -# reproducer appears to fail but actually never ran inside the sandbox at all): -openshell sandbox exec ai bash -c '...' -``` - -Issue #2592's first run hit this — wasted ~15 min before the maintainer noticed. Always use the `-n <name> -- <cmd>` form when the reproducer touches in-sandbox commands. - -**`brev exec` SSH-drop re-execution guard.** Brev's CLI silently retries from the top when the SSH connection drops mid-run, producing two parallel reproducer executions (we hit this on #2592 — one onboard process clobbered another's state, and both got billed). Use a sentinel file in the reproducer wrapper to make the script idempotent: - -```bash -# At the top of the reproducer wrapper script: -SENTINEL=~/.verify-stale-running -if [ -f "$SENTINEL" ]; then - echo "ERROR: another verify-stale run is in progress (sentinel: $SENTINEL)." - echo " If you're sure no other run is active, rm $SENTINEL and re-invoke." - exit 1 -fi -trap 'rm -f "$SENTINEL"' EXIT -touch "$SENTINEL" -``` - -The sentinel survives an SSH drop because it lives on the Brev box's filesystem; the trap removes it on script exit. A second `brev exec` invocation that tries to retry from the top will hit the sentinel and bail instead of double-running. - ---- - -### Step 8b: Run reproducer on baseline, compare to issue symptom - -If `./reproducer.sh` exists (verbatim from Step 6), run it. Otherwise synth on demand from the issue body (apply −30 penalty now, locked in for the rest of the run). - -**Interactive subcommand handling.** Many `nemoclaw onboard` / `nemoclaw configure` invocations prompt for input and will hang in a non-interactive shell. Auto-detect such subcommands in the script and apply, in order: - -1. Add `--non-interactive` if the version supports it. -2. Add `--dangerously-skip-prompts` (issue #2168 confirmed this exists for at least some Jetson paths). -3. Pre-feed answers via stdin: `printf 'yes\n\n\n' | nemoclaw onboard ...` - -If none work, route the script to Step 8c (synth-repro) so the LLM can rewrite it using non-interactive equivalents. - -```bash -brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh -brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcript.log -``` - -**Log-scraping (when `BUG_CLASS=log-only`).** Some bugs describe symptoms that show up in internal log files, not the reproducer's stdout/stderr — e.g., #1642 "see lots of error in openclaw log," #2611 "os.networkInterfaces guard errors." After running the reproducer, also pull the relevant logs from inside the sandbox and search them for the issue's symptom phrase: - -```bash -# Common NemoClaw / OpenClaw / OpenShell log paths inside the sandbox. -brev exec "$INSTANCE_NAME" "sg docker -c 'cat ~/.openclaw/logs/*.log /var/log/nemoclaw/*.log 2>/dev/null'" \ - | tee ./baseline-logs.log - -# Search the log capture for the issue's symptom phrase too, not just the transcript. -grep -F "<symptom phrase from issue body>" ./baseline-logs.log -``` - -For functional bugs the reproducer's stdout is sufficient; for log-only bugs the transcript may be clean but the log capture has the symptom. Both halves feed into the match rubric below. - -**Flake-detection retry.** Even for `functional` bugs, race-prone reproducers (TUI rendering, network policy negotiation, concurrent sandbox state) can produce inconsistent results. Run baseline three times if the first run shows the symptom inconsistently — same script, same env, just three back-to-back invocations. If the three runs disagree, that's signal: - -| 3-run baseline result | Verdict | -|---|---| -| All three reproduce the symptom | Strong baseline match → continue to 8d | -| All three are clean (no symptom) | Reproducer doesn't expose the bug on baseline → Step 8c synth-repro | -| Mixed (1 or 2 of 3 show the symptom) | Flake-prone reproducer. Note "flake suspected" in the comment; apply −25 to Step 9 score; downgrade `+50 latest clean` to `+25` because a clean latest run could just be the lucky path of an intermittent bug | - -Skip flake retry for `performance` and `rebuild-cycle` classes — those have their own multi-run rubrics in Steps 8e and 8f. - -**Match rubric.** LLM compares `baseline-transcript.log` to the issue's "Actual result" / error description. Match criteria, in order: - -1. **Exit code agrees** with what the issue describes (non-zero if issue describes a failure, zero if issue describes a wrong-output bug). Necessary but not sufficient. -2. **Symptom phrase match:** transcript contains a key error phrase from the issue (e.g., issue says `Permission denied on generate-openclaw-config.py`, transcript says `EACCES: permission denied, open '...generate-openclaw-config.py'` — semantic equivalence counts). -3. **Distinguish bug from infra noise:** generic network / DNS / auth errors don't count as a match unless the issue itself describes them. A bug about config parsing that fails at "could not resolve nvidia.com" is an infra failure, not a reproduction. - -**Fallback for issues without an explicit "Actual result" section.** Many bug reports describe a *behavioral* problem rather than a runtime error — e.g., "should default to a stable released version" (#1242), "configuration is not persisted across rebuilds" (#3030). These have no comparable error string. In that case: - -1. Use the issue's **full title + description** as the symptom signal. -2. Match if the reproducer's outcome **contradicts the issue's stated expected behavior** (or matches the stated wrong behavior). E.g., issue says "expected: stable release; actual: nightly", reproducer prints `nightly-build-2026.04.x` → that's a match. -3. If neither error string nor expected-behavior contradiction can be identified, route the script to Step 8c (synth-repro) — let the LLM produce a more diagnostic script that emits something testable. - -- **Match** → reproducer validated. Proceed to 8d. -- **No match** (silent pass, wrong error, infra noise, or no testable outcome): script has gaps. Proceed to 8c. - -### Step 8c: Synth-repro and retry on baseline - -LLM rewrites `./reproducer.sh` using the full issue context (description, environment, symptoms) **plus the baseline transcript** so it can react to what actually happened. Apply **−30 confidence penalty** (or keep it if 8b already applied it for the missing-verbatim case). - -```bash -brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh -brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcript-2.log -``` - -- **Match:** validated (with −30 baked in). Proceed to 8d. -- **Still no match:** mark `verify-inconclusive`. Post a comment that includes both reproducer attempts and both baseline transcripts with the message "couldn't establish a working reproducer for this bug on `$REPORTED_VERSION`." **Skip 8d** — there's nothing to verify on latest. - -### Step 8d: Install latest, run validated reproducer - -```bash -brev exec "$INSTANCE_NAME" "$RESET" -brev exec "$INSTANCE_NAME" " - if [ -f ~/.nvidia-api-key ]; then export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key); fi - curl -fsSL $INSTALL_URL | bash -" - -# Same resolved-version check as Step 8a — guard against env-var scoping or default fallthrough -# silently installing the wrong version. The latest install should resolve to $LATEST. -RESOLVED=$(brev exec "$INSTANCE_NAME" "bash -lc 'nemoclaw --version'" 2>&1 | tail -1) -echo "[verify-stale] latest requested: $LATEST; resolved: $RESOLVED" -case "$RESOLVED" in - *"$LATEST"*) ;; # match — proceed - *) echo "WARN: latest install resolved to '$RESOLVED' (expected match for $LATEST). Proceeding but flag in comment." ;; -esac - -brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh -brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./latest-transcript.log -``` - -If the install of **latest** fails (e.g. installer regression — see #3058 for a current example), this is an infra failure — see Step 11. Do not score or label the issue. - -If install succeeds, `latest-transcript.log` is the input to Step 9 scoring. - -For interactive debugging when something looks off: - -```bash -brev shell "$INSTANCE_NAME" -``` - ---- - -## Step 8d.5: Architectural-Drift Check - -Cross-version verification compares two moving targets: the reproducer assumes `$REPORTED_VERSION`'s tooling surface, and `$LATEST` may have rewritten the surface entirely. If the *tool* the reproducer relies on (CLI subcommand, output table, log file location) was reworked between the two tags, an "empty / clean output on latest" can mean either "bug fixed" OR "we're looking at a deprecated tracking surface." Without this check, the latter silently registers as the former — a class of false positive. - -**Detection** — pickaxe the diff between tags for the reproducer's tool name and watch for the CLI itself being touched, not just its consumers: - -```bash -# Extract the primary verification command from the reproducer (e.g. "openshell forward list"). -TOOL=$(grep -oE '\b(openshell|nemoclaw)[[:space:]]+[a-z-]+' reproducer.sh | sort -u) - -# Pickaxe each tool name across the version range. -for t in $TOOL; do - echo "=== drift check: $t ===" - git log "$REPORTED_VERSION".."$LATEST" -S"$t" --oneline -- src/ bin/ nemoclaw/src/ 2>&1 | head -5 -done -``` - -If a tool is touched, drift is suspected. - -**Multi-axis verification** — when drift is suspected, do not rely on the reproducer's expected output alone. Pick OS-level surfaces that would show the buggy state regardless of which CLI tracks it. For port-forwarding bugs (the #2007 case), the canonical five-axis pattern: - -| # | Surface | Command | -|---|---|---| -| 1 | Reproducer's stated check | as written in the issue body | -| 2 | Host TCP listeners | `sudo ss -tlnp` | -| 3 | iptables NAT redirects | `sudo iptables -t nat -L -n` | -| 4 | Docker port mappings | `docker ps --format '{{.Names}} {{.Ports}}'` | -| 5 | Active SSH tunnels | `ps -ef \| grep 'ssh.*-L'` | - -Adapt the axes to the bug class. For filesystem bugs: `find`, `lsattr`, `stat`. For network policy bugs: `iptables -L`, container netns, gateway logs. The principle is the same — pick at least three independent surfaces that would each independently show the buggy state if it were present. - -**Action when drift is suspected:** - -- Run the multi-axis pattern after Step 8d's reproducer. -- The verdict requires **every relevant axis to be clean** — not just the reproducer's surface — before claiming `fixed-on-latest`. -- Quote the multi-axis evidence in the Step 10 comment as a table; this is exactly what makes "fixed" defensible when the original tooling no longer reflects the underlying behavior. -- If any axis still shows the buggy state, the bug is NOT fixed even if the reproducer's surface is clean. Escalate to "still reproduces" (Step 9 special case). - -**When drift is NOT suspected** (the reproducer's tool is unchanged in the version range): the reproducer's expected output is sufficient, no multi-axis verification needed. - ---- - -## Step 8e: Performance-Bug Verification (when `BUG_CLASS=performance`) - -Performance bugs (#2598 "10s P50", #2600 "hangs ~2 min", #2733 Ollama tool-call leak over time) can't be answered by the standard exit-code + symptom-phrase rubric — one clean reproducer run doesn't tell you the p50 budget is met; one slow run doesn't tell you the bug still reproduces. Replace Step 8b's match with a measurement-and-distribution rubric: - -1. **Parse the SLA from the issue body.** Extract numeric latency thresholds: `10s P50`, `200ms`, `under 5 seconds`, `~2 min`. Save as `SLA_P50_MS`, `SLA_P90_MS`, etc. If no numeric SLA is in the body, route to Step 8c synth-repro to ask the reporter (via comment) for one — without a target, the verdict is undefined. -2. **Run the reproducer N=10 times** on each side (baseline + latest), capturing per-run latency: - - ```bash - for i in $(seq 1 10); do - /usr/bin/time -f '%e' bash ~/reproducer.sh >/dev/null 2>>./latest-perf.log - done - ``` - -3. **Compute p50 and p90** for both sides. `sort -n ./latest-perf.log | awk 'NR==5'` for p50 of 10 runs. -4. **Match rubric:** - - Latest's p50 within the SLA AND baseline's p50 outside the SLA → bug fixed; same Step 9 scoring (subject to baseline-validation gate). - - Latest's p50 outside the SLA → bug still reproduces (Step 9 special case). - - Latest p50 within SLA AND baseline p50 also within SLA → reproducer doesn't actually exercise the bug; route to Step 8c synth-repro. - -**Hardware-substitution caveat.** Performance numbers are silicon-dependent. When the issue is `Platform: DGX Spark` or `Platform: GB10` and we're measuring on a Brev x86 GPU SKU, the comment must say so explicitly: a Brev p50 of 1.5s on a `H100` does not prove the DGX Spark p50 is fixed. Cap the score at 60 unless the bug is clearly silicon-independent (e.g. an algorithmic regression in user-space JS that would manifest the same on any silicon). - ---- - -## Step 8f: Rebuild-Cycle Verification (when `BUG_CLASS=rebuild-cycle`) - -Rebuild-cycle bugs (#2701 "Pod recreate wipes `/tmp/nemoclaw-proxy-env.sh`," issues describing "configuration is not persisted across rebuilds") only manifest when sandbox state crosses a destroy/recreate boundary. A single onboard run can't trigger the symptom. Replace Step 8b's match with a run-rebuild-rerun harness: - -1. **First onboard.** Run the reproducer once to establish initial state. Capture relevant artifacts (config files, env vars, sandbox metadata) — the issue body usually names what should persist: - - ```bash - brev exec "$INSTANCE_NAME" "sg docker -c 'cat <files-mentioned-in-issue> 2>&1'" | tee ./pre-rebuild.log - ``` - -2. **Trigger the rebuild.** Use `nemoclaw destroy --all --force` followed by `nemoclaw onboard` with the same env vars. Do NOT comprehensive-reset between (the point is to test the destroy/recreate, not start from scratch). - -3. **Re-capture the same artifacts** post-rebuild: - - ```bash - brev exec "$INSTANCE_NAME" "sg docker -c 'cat <same-files> 2>&1'" | tee ./post-rebuild.log - ``` - -4. **Diff and match.** The bug is "X gets wiped / changes / regresses across rebuild." Compare pre-rebuild vs post-rebuild captures to the issue's expected behavior: - - Pre and post agree (artifact preserved) AND issue says it should be preserved → bug fixed - - Pre and post differ (artifact wiped) AND issue says it gets wiped → bug still reproduces - - Pre and post agree AND issue says it gets wiped → reproducer doesn't exercise the bug; Step 8c synth-repro - -The harness still uses Step 9's scoring framework — `+50 latest clean (artifact preserved)`, etc. — but the "what gets compared" axis is the diff, not the symptom phrase. - ---- - -## Step 8.5: Detect "Behavior Changed by Design" - -Before scoring, check whether the symptom is intentional. Some bugs are filed against behavior that was **deliberately changed or removed** in a merged PR — running the standard rubric on these produces misleading verdicts. The symptom "still reproduces" but the right answer is "won't fix, see PR #X." Issue #2791 is the prototype: `config set` was removed in PR #2227, the reporter tested a version that already had it gone, and a standard rubric run would have buried that context under a low-confidence `verify-inconclusive` label. - -This step is split into substeps so the rigor is mechanical, not optional. Every claim in the final comment must be backed by a verifiable evidence block — a comment URL with quoted phrase, a commit SHA with diff range, or a grep command with its actual output. Hand-wavy claims fail Step 8.5d's self-verification pass and force a bail to `verify-inconclusive`. - -### Step 8.5a: Run signal detection - -Any single signal is sufficient to trigger the by-design branch. - -**Signal 1 — Maintainer attribution in comments.** Any comment by an author with `authorAssociation` of `MEMBER`, `OWNER`, or `COLLABORATOR` matches `removed in #\d+`, `removed in [Pp][Rr] ?#\d+`, `by design`, `wontfix`, `won't fix`, `not a bug`, or `intentional`. - -```bash -gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ - --jq '.comments[] - | select(.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR") - | select(.body | test("removed in #\\d+|by design|wontfix|won.t fix|not a bug|intentional"; "i")) - | {url, author: .author.login, body}' -``` - -Capture for evidence: comment URL + author login + the exact quoted phrase. - -**Signal 2 — Removal commit in range.** A commit between the reported version and `$LATEST` deletes the symbol implicated by the reproducer (CLI subcommand, function, flag). The commit subject does NOT need to mention "remove" / "delete" — many removals ride into a `refactor(...)` or `feat(...)` commit (e.g. PR #2227 removed `--dangerously-skip-permissions` under a `refactor(sandbox): ...` subject). Use git's pickaxe to find the responsible commit by content: - -```bash -# Pickaxe: list every commit whose diff changes the count of <symbol> occurrences. -# Reverse order so the earliest removal commit lands first in the list. -git log "$REPORTED_VERSION".."$LATEST" -S'<symbol>' --reverse --oneline -- src/ bin/ nemoclaw/src/ - -# Subject-keyword narrowing is only a SUPPLEMENTARY lookup — useful when the -# pickaxe returns many commits and you want to focus on the obviously-removal one. -git log "$REPORTED_VERSION".."$LATEST" --grep='remove\|delete\|drop\|deprecate' -i --oneline - -# For each candidate, confirm the diff actually deletes the symbol (not just renames or moves it). -git log -p <candidate-sha> -- src/ bin/ nemoclaw/src/ | grep -nE '^-.*\b<symbol>\b' -``` - -Capture for evidence: commit SHA + each `file:line` block of deletions touching the symbol. Note the commit's actual subject — don't assume it says "remove." - -**Signal 3 — Symbol absent in both reported version and latest.** The implicated symbol (e.g. `config set`) is not present in either tag's source tree — meaning the responsible change landed before the version the reporter tested. This is the #2791 case. - -```bash -git grep -n "<symbol>" "$REPORTED_VERSION" -- src/ bin/ nemoclaw/ # expect: zero matches (or shim-only — see sub-case) -git grep -n "<symbol>" "$LATEST" -- src/ bin/ nemoclaw/ # expect: zero matches (or shim-only) -``` - -Capture for evidence: both grep commands and their (empty) outputs. - -**Sub-case for signals 2 and 3 — vestigial deprecation shims.** It's common for a removed symbol to survive in latest *only* as a deprecation message (e.g., a CLI subcommand that prints `"--<flag> was removed; use <X> instead"` and exits non-zero). When a grep returns matches in latest, inspect each `file:line`. If every match is a deprecation stub with no functional effect on the bug-as-filed, signal 2 or 3 still fires; record the shim locations and behavior as a separate evidence block. Do not silently treat shims as functional code, and do not silently treat them as absence. - -### Step 8.5b: Pre-check related failure modes - -A by-design verdict says "the bug *as filed* can't reproduce." It does NOT say "every bug shaped like this is fixed." Before drafting the comment, search latest's source for code paths that could still produce the issue's described **symptom** (not the literal removed flag/symbol — the symptom). - -```bash -# Use the issue's symptom keywords, not the removed symbol. -git grep -nE "<symptom-keyword-1>|<symptom-keyword-2>" "$LATEST" -- src/ nemoclaw/src/ -``` - -For #2168 the literal flag is `--dangerously-skip-permissions`, but the symptom is "sandbox created but not registered in CLI." Grepping for `register.*[Ss]andbox`, the readiness-gate / cleanup-failure path in `src/lib/onboard.ts` surfaces as a related-but-different way to produce an orphan sandbox. - -If a related failure mode is found, the by-design comment MUST include a "What's not literally the same bug" section that names it with `file:line`. Don't suppress the call-out by claiming "the symptom is impossible" when the symptom can be reached via a different path. - -### Step 8.5c: Check existing test coverage - -Search the repo for tests that exercise the NEW intended workflow (the one that replaced the removed symbol). Citing them strengthens the comment from "trust me, it was removed" to "the new workflow is exercised by these tests." - -```bash -git grep -lnE "<new-workflow-keyword>" -- test/ nemoclaw/src/ 2>/dev/null | head -5 -``` - -Cite at most three concrete test paths. If none exist, omit the section — do not invent paths. - -### Step 8.5d: Self-verification pass before posting - -Two passes, both required. - -**Evidence pass.** Re-run every grep / git / `gh` command cited in the evidence blocks. If any cited `file:line`, commit SHA, or quoted output doesn't reproduce on a fresh invocation, **stop and revise** — or bail to `verify-inconclusive` if the discrepancy can't be resolved. - -**Link pass.** Resolve at least one rendered markdown link from each section that has them — `What's structurally fixed`, `Vestigial references`, `Existing CI coverage`. Use `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 content if the path exists at the tag, 404 otherwise) or `curl -fsI <blob-url>` (returns 200 if the blob renders). A broken link is worse than no link — it suggests verification work that didn't actually happen. - -The cost of an incorrect "I checked and X is gone" claim in a public comment, or a 404 on a citation, is higher than spending a minute re-checking. This step exists because LLMs can confidently overstate and confidently invent paths; mechanical re-verification catches both. - -### Step 8.5e: If any signal fires - -- **Skip the Step 9 score table** entirely. The "exit 0 + expected output" axis doesn't apply when the expected output is no longer the contract. -- **Skip Brev provisioning** if the signal fires before Step 7 — a remote run would just confirm what static analysis already proved. (Signals 2 and 3 can run as soon as the reported version is parsed in Step 4.) -- **Apply label `status: wont-fix`** (the existing repo label — quote it on the CLI: `gh issue edit <num> --add-label "status: wont-fix"`). It's already in the Step 3 issue-type skip list, so a labelled issue is automatically excluded from future runs without needing a separate idempotency clause. -- **Use the by-design comment template below** instead of the standard Step 10 template. -- **@-mention the reporter** so they can object if the framing is wrong. -- **Never auto-close.** A maintainer pulls the trigger, same as the other label paths. - -### By-design comment template - -Mandatory sections in this order. Omit only the sections explicitly noted as omittable. - -**Tag-anchoring + linking rule.** Every `file:line` citation, commit SHA, and test-path reference in the rendered comment MUST be a clickable markdown link to the verified-on tag (e.g., `v0.0.35`), not the maintainer's working `HEAD`. Lines drift between tags and main; tag-anchored links keep the citations reproducible by anyone reading the comment months later. Bare paths force the reader to navigate manually — that's a usability bug, not a stylistic preference. - -Use these exact link formats: - -- File only: `[src/lib/onboard.ts](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/onboard.ts)` -- File:line: `[src/lib/onboard.ts:4965](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/onboard.ts#L4965)` -- File:line-range: `[src/lib/commands/sandbox/connect.ts:25-31](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/commands/sandbox/connect.ts#L25-L31)` -- Commit SHA: `[5956a61](https://github.com/NVIDIA/NemoClaw/commit/5956a612e18047b9ab85b3a7e89f6b5dedb29190)` — short SHA as the link text, full SHA in the URL -- Test file: `[test/e2e/test-double-onboard.sh](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/test/e2e/test-double-onboard.sh)` -- PR/issue references: bare `#NNNN` works — GitHub auto-links these in comments on the same repo, no manual URL needed. - -When greping for evidence, use `git grep -n "<symbol>" "$LATEST" -- ...` so the line numbers match the tagged blob. Then construct each link from `<file path> + verified-on tag + line number`. - -The Step 8.5d self-verification pass MUST resolve at least one rendered link (e.g., `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=v0.0.35` or a `curl -fsI` to the blob URL) and confirm it returns the expected file. A broken link defeats the purpose of including the citation. If any link fails to resolve, fix it or bail to `verify-inconclusive`. - -````markdown -## Stale-issue verification — behavior is by-design - -**Reported on:** v0.0.<X> -**Verified on:** v0.0.<Y> (PR #<NNNN> first shipped in v0.0.<Z>) -**Verification mode:** static analysis at the verified-on tag — no runtime reproduction. Step 8.5 by-design short-circuits Brev provisioning because the responsible code change is already proven by the diff between `$REPORTED_VERSION` and `$LATEST`. -**Outcome:** symptom reproduces against the reproducer as filed, but the implicated behavior was intentionally changed. - -### What's structurally fixed - -- `<file:line>` — `<one-sentence summary of the change at that location>` -- `<file:line>` — `<…>` - -The new workflow is `<one-sentence: how to do what the user was trying to do>`. - -### Vestigial references - -- `<file:line>` — `<deprecation behavior: e.g. "prints '--<flag> was removed; use <X> instead' and exits 1; no functional effect">` - -(Omit this section entirely when the symbol is fully gone with no surviving stubs.) - -### What's not literally the same bug - -`<one-sentence acknowledgement of the related failure mode found in Step 8.5b, with file:line>` — OR — `None. The symptom requires the removed symbol; no related code path produces it on latest.` - -### Existing CI coverage - -- `<test/path/file>` — `<one-sentence: what this test demonstrates about the new workflow>` - -(Omit when no direct test exists. Do not invent paths.) - -### Recommendation - -@<reporter> — please confirm the by-design framing is correct (the implicated `<symbol>` was intentionally removed, the original reproducer can no longer execute) and close as "won't fix / by design" if you agree. If a related symptom (e.g. `<related failure mode from above>`) is hitting you on ≥ v0.0.<Z>, please file a fresh issue with a v0.0.<Z>+ reproducer. - -`<NVBugs cross-ref line — see below>` - -<!-- nemoclaw-verify-stale v1 YYYY-MM-DD --> -```` - -**NVBugs cross-ref line.** If `NVBUGS_REF` was set in Step 4, append: - -> NVBugs<NVBUGS_REF without brackets> will need a separate update; closing this GitHub issue won't propagate. - -Otherwise omit the sentence. - -**If no signal fires:** continue to Step 9 normally. - ---- - -## Step 9: Score Confidence - -Start at 0. Apply each rule that fires. - -| Signal | Delta | -|---|---| -| Reproducer ran cleanly on **latest** (8d), exit 0, no bug symptom observed | +50 | -| Commits between reported version and `$LATEST` touch the implicated component (see "Path extraction" below) | +25 | -| A merged PR mentions this issue number or its symptom (see "PR search" below) | +25 | -| Reproducer was LLM-synthesized at any point (Step 8b synth or Step 8c retry) | −30 | -| Any partial error, warning, or flaky behavior in the latest run (8d) | −50 | - -Total is clamped to `[0, 100]`. - -### Path extraction (for the +25 commits signal) - -The skill needs to know *which* path to `git log v<reported>..$LATEST -- <path>` against. Apply in order, stop at the first that yields a non-empty path: - -1. **Stack trace / file path mentions in the issue body.** Grep the body for absolute paths under known install roots, then map to repo paths: - - `/usr/local/lib/nemoclaw/<rel>` → `<rel>` in repo (e.g., `scripts/generate-openclaw-config.py`) - - `/usr/local/bin/nemoclaw*` → `bin/` - - `~/.nemoclaw/<rel>` → most often runtime state, drop unless the bug is config-related → `src/lib/config/` - - In-repo paths (e.g., `bin/lib/policies.js` mentioned literally) → use as-is -2. **Component-label-to-directory map.** Pick the first match. Paths verified against the current repo layout — drop any path that doesn't exist on the tag at `$LATEST` rather than passing it to `git log`. - - `NemoClaw CLI` → `bin/`, `src/lib/`, `nemoclaw/src/commands/` - - `Sandbox` → `nemoclaw/src/blueprint/`, `nemoclaw-blueprint/` - - `OpenShell` → cross-repo (lives at `github.com/NVIDIA/OpenShell`, not in this repo). Skip the +25 signal for OpenShell-only issues; cross-repo `git log` is out of v1 scope. - - `Docker` → `Dockerfile`, `Dockerfile.base`, `scripts/install-openshell.sh`, `scripts/install.sh` - - `Getting Started` → `docs/`, `scripts/install.sh` - - `Integration: <X>` — no `src/lib/integrations/` exists in this repo. Skip the +25 signal for integration-component issues unless source 1 (file paths in body) yielded a path. -3. **Title keywords.** "policy" → `nemoclaw-blueprint/policies/`, `nemoclaw/src/blueprint/`. "inference" → `docs/inference/` is docs-only; skip the +25 signal unless source 1 surfaces actual code paths. - -If none of the above produces a path, **skip the +25 signal entirely** rather than guessing. Floating the +25 on every issue would inflate scores meaninglessly. - -### PR search (for the +25 PR signal) - -```bash -# Direct issue-number reference (covers most cases — "fixes #2861" etc.) -DIRECT_REF=$(gh pr list --repo NVIDIA/NemoClaw --state merged \ - --search "$ISSUE_NUMBER" \ - --json number,title,mergedAt,body \ - -q "[.[] | select((.body + \" \" + .title) | test(\"#$ISSUE_NUMBER\\\\b\"))]") - -# Symptom-phrase fallback (only if direct reference returns nothing) -if [ -z "$DIRECT_REF" ] || [ "$DIRECT_REF" = "[]" ]; then - SYMPTOM=$(extract first key error/symptom phrase from issue body, ~3-6 words) - SYMPTOM_REF=$(gh pr list --repo NVIDIA/NemoClaw --state merged \ - --search "\"$SYMPTOM\"" \ - --json number,title,mergedAt) -fi -``` - -Apply +25 if either query returns at least one PR with `mergedAt` strictly after the tag date of `$REPORTED_VERSION` (look up via `git log -1 --format=%cI v$REPORTED_VERSION`). PRs merged before the reporter even filed the issue can't have fixed it. - -If neither query returns anything, **skip the +25 signal**. - -**Baseline-validation gating.** The +50 weight assumes the reproducer was *validated* — i.e., it produced the bug symptom on baseline (Step 8b/8c match). If `BASELINE_INSTALL_FAILED=1` (Step 8a fall-through, baseline pass skipped — including the sandbox-build-rot case from Step 11), the +50 still applies but **cap the total at 84**. Corroboration signals (commits-touched-area, PR-mention) still raise the score within the cap but cannot lift it above 84. Without runtime baseline confirmation we don't have enough on our own to claim ≥85 — the cap forces the verdict into the 60–84 band where the reporter is asked to confirm. The previous draft of this rule had an "unless commits-touched OR PR-mention also fires" escape hatch that let inferred fix evidence bypass the cap entirely; that produced a misleading 100/100 on the #2007 e2e run despite zero baseline confirmation, and was tightened here. - -**Action (when latest run was clean — bug not reproduced):** - -| Score | Label | Comment | -|---|---|---| -| ≥85 | `fixed-on-latest` | Evidence-rich, no @-mention. | -| 60–84 | `fixed-on-latest` | Evidence-rich, **@-mention the original reporter** to confirm. | -| <60 | `verify-inconclusive` | Short, honest "couldn't verify" explanation. | - -**Special case: latest output matches the issue symptom (bug still reproduces on latest).** - -This is not a flake — the skill positively confirmed the bug is still live. Don't apply the +50 weight (the bug isn't fixed) and skip the score table entirely. - -- Post a "still reproduces on latest" comment with both transcripts. -- Apply **no label**. -- Include the marker `<!-- nemoclaw-verify-stale v1 YYYY-MM-DD -->` with today's date so the candidate filter applies the 7-day TTL (Step 3 idempotency). -- Next weekly run picks the issue back up after the TTL — if the bug gets fixed in the meantime, that run catches it. - -The skill **never closes issues** in any branch. A maintainer pulls that trigger after reviewing the label and comment. - ---- - -## Step 10: Compose and Post the Comment - -**Redaction pass before posting.** Run on **every** chunk of text quoted in the comment — issue body excerpts, baseline transcript, latest transcript, synth-repro scripts. Replace each match with `[REDACTED]`. The transcripts especially leak — they include full stdout/stderr from real installs and runs. - -**HTML → text pre-pass for issue body excerpts.** NV QA bodies are HTML; tokens nested in `<pre>` tags or HTML attributes (e.g. `<a href="https://user:tok@host/...">`) slip past the regex patterns below if the input still has tags. Convert to plain text first, then redact: - -```bash -TEXT=$(printf '%s' "$BODY_EXCERPT" | python3 -c ' -import html, re, sys -b = sys.stdin.read() -b = re.sub(r"<br\s*/?>", "\n", b) -b = re.sub(r"</?(p|div|tr|td|th|li|pre)[^>]*>", "\n", b) -b = re.sub(r"<[^>]+>", "", b) -print(html.unescape(b)) -') -# Now apply the regex table below to $TEXT. -``` - -Transcripts and synth-repro scripts are already plain text and skip the pre-pass. - -**Order matters and the table below is in execution order.** Longest, most-specific patterns first; generic catchalls last. Otherwise the catchall masks specific matches and you lose track of what was actually redacted (JWT vs session blob vs random base64). - -| # | Pattern | Targets | -|---|---|---| -| 1 | `eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}` | JWT tokens | -| 2 | `gh[pousr]_[A-Za-z0-9]{36,}` | GitHub PATs / install tokens | -| 3 | `(?i)nvapi-[A-Za-z0-9_-]{20,}` | NVIDIA API keys (NIM / build.nvidia.com) | -| 4 | `AKIA[0-9A-Z]{16}` | AWS access key IDs | -| 5 | `(?i)aws_secret_access_key\s*=\s*\S+` | AWS secret keys | -| 6 | `(?i)authorization:\s*\S+` | HTTP auth headers (often Bearer + JWT) | -| 7 | URLs containing `@` before the host (e.g., `https://user:pw@host/...`) | Basic-auth credentials in URLs | -| 8 | `(?i)(token\|secret\|password\|api[_-]?key\|bearer)[^\n]*[:=][^\n]*` | Inline credentials in env/config/log output | -| 9 | `\b\w+\.(nvidia\.internal\|nv-internal\.com\|nvidia\.dev)\b` | Internal hostnames (extend list per team) | -| 10 | `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}` | Email addresses (PII) | -| 11 | `\b[A-Za-z0-9+/]{60,}={0,2}\b` | Long base64 blobs (likely keys/sessions; tune length to taste — too short hits legit data) | - -**File paths under the reporter's home directory** (`/Users/<name>/`, `/home/<name>/`) → replace with `~/`. Run last; catches incidental username PII. - -**Comment authoring principle.** Every section in a rendered comment must either change a reader's mind about the verdict, or be cut. Word counts follow from that — **300 is a hard ceiling** for the main verdicts (fixed-on-latest, wontfix). Simple cases (clear PR ref, deterministic check) land under 200. The principle generalizes: comments posted by this skill compete for a maintainer's attention against every other in-flight thread, and "AI-slop" prose — architectural sidebars, file:line citations the maintainer can find via the PR ref, bare-output reproductions when the load-bearing evidence is elsewhere, "if this verification is wrong, please reopen…" boilerplate — actively reduces the comment's signal-to-noise ratio. - -**For each section in a draft, ask: would the maintainer reach a different conclusion *without* this section? If no, delete.** Lessons accumulated from real runs: - -- **#2007 first draft (~750 words):** had a multi-paragraph "Architectural notes for QA reference" section that didn't change the verdict. Cut → 371 words. -- **#2604 first three drafts:** wavered between fixed-on-latest, still-reproduces, and by-design across iterations because each draft padded the verdict with prose that didn't ground it. Final 190-word draft cut a maintainer-note sidebar about platform attribution, a bare-status output reproduction, and a file:line citation of the source — none affected the verdict, all were AI-slop padding. Rule learned: **before drafting any prose, name the verdict in one sentence; if a section doesn't directly support that one sentence, cut it before writing it.** - -**Per-verdict length defaults:** - -| Verdict | Target | Rationale | -|---|---|---| -| `fixed-on-latest` | **200–300 words** | Header + evidence + verdict + @-mention. Add hardware-substitution caveat or related-failure-mode section only if they shift the maintainer's read. If you're past 300, you're padding. | -| `wontfix` (by-design) | **200–300 words** | Structurally-fixed + vestigial + what's-not-the-same-bug, each one to two sentences max. The PR ref carries the detail; the comment carries the verdict. | -| `verify-inconclusive` | 100–200 words | One paragraph naming what the skill couldn't establish. No transcripts beyond a single quoted line. | -| **Still-reproduces (no label)** | **30–80 words** | The reporter already has the symptom; the maintainer can see the issue is open. The skill is just confirming + setting the TTL marker. **No transcripts** (the issue body has them), **no @-mention** (the reporter knows their bug is real), **no architectural prose**. One sentence stating "skill ran reproducer on `<latest>`, symptom still present" + one sentence on any partial-fix PR if relevant + marker. That's it. | - -**Cut, by default:** - -- Maintainer-note sidebars about labels / platform attribution unrelated to the bug surface. -- Bare-output reproductions when the load-bearing evidence is in a different command's output. -- File:line citations of source code already findable via the cited PR. -- Closing "if this verification is wrong, please reopen…" boilerplate. -- Redundant verbal framing of what the evidence already shows ("the table above proves…"). -- "Verification mode" pleasantries beyond one factual line. - -**Mandatory cap caveat.** When the score is capped (Step 9 baseline-validation gating, or any Step 11 degraded-mode path), the rendered Verdict section must include a one-line caveat naming the cap and the reason. Example: `Capped at 84 because Step 9's baseline-validation gate did not run (sandbox-build rot on v0.0.18: Dockerfile symlink layer removed by #2227).` Don't make readers reverse-engineer why the score didn't go higher — name it. - -**Mandatory hardware-substitution caveat.** When the issue carries `Platform: DGX Spark` or `Platform: GB10` and Step 7 provisioned a Brev SKU that is not the same silicon (Brev's stoppable GPU catalog is x86 + discrete H100/A100/L40S/T4 — not Grace Hopper / GB10 unified-memory ARM64), the rendered comment must include a one-line "Hardware substitution" note. Example: `Hardware substitution: verified on Brev n1-standard-4:nvidia-tesla-t4 (x86_64 + T4) as a substitute for the reporter's DGX Spark (ARM64 + GB10). For silicon-shape bugs (perf, memory architecture, drivers) this is not a faithful repro — please confirm on actual DGX Spark.` This goes in the metadata block right after `Verification mode:` so it's visible at the top, not buried in the analysis. - -**Mandatory `Verification mode` header line.** All three templates below include a `**Verification mode:**` line in the metadata block, naming what we did and didn't actually run (e.g., "runtime reproduction on Brev <SKU>; baseline + latest both installed and run" for the standard template; "static analysis at the verified-on tag — no runtime reproduction" for the by-design template; "runtime reproduction on Brev <SKU>; bug confirmed live on latest" for still-reproduces). Reader should never have to guess whether the verdict came from real install logs or from static analysis. - -**Link-pass self-verification (all templates).** Same rule as Step 8.5d's link pass, applied to every template. Resolve at least one rendered markdown link from each section that has them (`What's structurally fixed` / `Vestigial references` / `Existing CI coverage` for by-design; `Relevant changes since` / transcript code-anchor citations for the standard template) via `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 if path exists at tag, 404 otherwise) or `curl -fsI <blob-url>`. A 404 on a citation in the rendered comment is worse than no citation — it advertises verification work that didn't actually happen. If any link fails to resolve, fix it or bail to `verify-inconclusive`. - -**Mandatory closing block — reporter @-mention with confirmation language.** Every template below ends with an explicit @-mention of the original reporter using this exact shape: - -> @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. - -The skill cannot independently confirm a closed-as-fixed verdict — only the reporter knows whether their original symptom is gone in their environment. The @-mention is what converts a "skill says it's fixed" claim into actionable confirmation work for QA. Customize `<Z>` per case (the version that shipped the fix or `$LATEST`), but never omit the line. - -**Mandatory unanswered-question prefix and dual @-mention.** When Step 3 sets `UNANSWERED_MAINT_LOGIN` (a maintainer's question is older than 7 days and the reporter never replied), the verdict comment changes shape in two places: - -1. **Prepend a lead paragraph** as the very first line of the body, before the `## Stale-issue verification` heading. The lead paragraph is a single line: - - ```text - [@UNANSWERED_MAINT_LOGIN's comment](UNANSWERED_MAINT_URL) from UNANSWERED_MAINT_DATE is still unanswered. Posting independent verification below to unstick the thread. - ``` - - …with the bracketed variables expanded from the values exported by Step 3. - -2. **Replace the closing reporter-only @-mention with a dual @-mention** that names BOTH the maintainer (acknowledging the open question) and the reporter (per the standard confirmation pattern): - - > @\<UNANSWERED_MAINT_LOGIN\> — flagging that your question above is still open; the verification below may answer it. @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. - -This applies to all three templates (fixed, still-reproduces, by-design). The skill becomes the *unsticking voice* on a thread that has gone quiet — never a clueless interruption when discussion is fresh (Step 3 already filtered the within-7-day case). - -**Comment template (fixed / inconclusive — bug not reproduced on latest):** - -````markdown -## Stale-issue verification — automated - -**Reported on:** v0.0.31 -**Verified on:** v0.0.34 (commit abc1234) -**Verification mode:** runtime reproduction on Brev `<instance-class>` — baseline (v0.0.31) and latest (v0.0.34) both installed and run; comparison made on the captured transcripts. (Or: "runtime reproduction on Brev `<instance-class>` — baseline-install-skipped (`.openclaw-data` rot, see Step 11), latest-only run; verdict capped at 84.") -**Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 / <CUDA version if GPU> - -### Baseline (reported version) - -- Install: succeeded · skipped (install rotted) -- Reproducer: extracted verbatim · synthesized (−30 penalty) -- Result: bug symptom matched (validated) · could not validate (skipped Step 8c gate) - -<details><summary>Baseline transcript</summary> - -```text -<full baseline transcript> -``` - -</details> - -### Latest - -- Install: succeeded -- Result: not reproducible — clean run, no bug symptom observed - -<details><summary>Latest transcript</summary> - -```text -<full latest transcript> -``` - -</details> - -### Verdict - -**Confidence:** 88 / 100. Labelling `fixed-on-latest`. - -<details><summary>Relevant changes since v0.0.31</summary> - -- abc1234 — fix: <commit subject> -- def5678 — refactor: <commit subject> - -</details> - -@<reporter> — please confirm the symptom is gone on a recent build (≥ v0.0.<Z>) and reopen with a fresh reproducer if you observe otherwise. - -<!-- nemoclaw-verify-stale v1 2026-05-12 --> -```` - -**Comment template (still reproduces — Step 9 special case):** - -````markdown -## Stale-issue verification — still reproducible - -**Reported on:** v0.0.31 -**Verified on:** v0.0.34 (commit abc1234) -**Verification mode:** runtime reproduction on Brev `<instance-class>` — baseline confirmed the symptom matches the issue; latest (v0.0.34) also produced the symptom. Bug is still live. -**Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 - -The skill ran the reported reproducer on v0.0.34 and observed the same bug symptom described in this issue. The bug is still live. - -No label applied. Will re-verify automatically next weekly run; if a fix lands in the interim, the next pass catches it. - -@<reporter> — please confirm the symptom still matches your observation on v0.0.<Y> and reopen with any updated reproducer or environment details if it has shifted. - -<details><summary>Baseline transcript (validated reproducer)</summary> - -```text -<baseline transcript> -``` - -</details> - -<details><summary>Latest transcript (bug still observed)</summary> - -```text -<latest transcript> -``` - -</details> - -<!-- nemoclaw-verify-stale v1 2026-05-12 --> -```` - -The trailing HTML comment is the **idempotency marker** Step 3 looks for. Always include today's date in `YYYY-MM-DD` format so the candidate filter can apply the 7-day TTL. - -**Post the comment and apply the label:** - -```bash -gh issue comment "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --body-file comment.md -gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "fixed-on-latest" -# or for <60: -# gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "verify-inconclusive" -``` - ---- - -## Step 11: Infra Failure Handling - -Two different failure types, two different responses. - -**Latest-install failure** (Step 8d) or reuse-check / provisioning / harness errors: hard infra failure. - -- Print the error. -- Apply **no label** — infra failures must not pollute the verification record. -- Post a short comment **only if explicitly requested by the invoking user**. Default is silent move-on. -- Continue to the next candidate in batch mode. - -The next weekly run retries naturally. - -**Baseline-install failure** (Step 8a, reported version won't install on a modern image): not a hard failure — degraded mode. - -- Set `BASELINE_INSTALL_FAILED=1`, skip 8b/8c, jump to 8d. -- Step 9 applies the score cap (max 84) — corroboration signals raise the score within the cap but cannot lift past it. -- Note "baseline-install-skipped" in the final comment so a reviewer knows the verification ran without the script-validation gate. - -**Baseline-build failure** (Step 8a binary install succeeded, but the in-image `Dockerfile` build during sandbox creation failed on a layer that was structurally removed in a later release): also degraded mode, distinct from binary install rot. Surfaced during the #2007 e2e run on v0.0.18 (`/sandbox/.openclaw-data/workspace/media` symlink layer, removed entirely by #2227). - -- Set `BASELINE_INSTALL_FAILED=1` (same flag — Step 9's cap-at-84 rule keys off it regardless of which phase rotted). -- Skip 8b/8c, jump to 8d. -- Note "baseline-build-skipped" in the final comment with the specific failing layer/file so a reviewer can see *why* the v0.0.X image no longer builds (the why is usually a follow-on PR that removed the rotted layer). -- Do not retry the build with a patched Dockerfile — that breaks faithfulness. We're claiming "couldn't independently re-trigger the original symptom on baseline," not "we made the old version work somehow." - -Both baseline-rot variants share the same downstream effect: Step 9 cap, Step 10 caveat, @-mention reporter to confirm. Distinguishing them in the comment helps a reviewer understand the failure mode without re-running. - -This degradation is expected — old releases rot at multiple phases (binary installer URL drift, base-image dependencies vanish, in-image Dockerfile layers get removed by structural refactors). We still want to extract whatever signal we can from the latest run plus PR/commit evidence, just at a more conservative confidence ceiling. - -**Empirical reality after two e2e runs:** baseline-build-rot is the **dominant** failure mode for any reported version more than ~5–7 patches behind, not an edge case. Both #2007 (v0.0.18, 17 patches behind) and #2592 (v0.0.28, 7 patches behind) hit it. The cap-at-84 with reporter @-mention is the **modal** verdict shape for stale-issue verification, not the exception. Reframe expectations accordingly: - -- For issues reported >5 patches behind `$LATEST`, plan for the cap-at-84 path. Pre-flight (PR-search, pickaxe) carries more weight than baseline runtime evidence. -- For issues reported within 1–4 patches of `$LATEST`, baseline is more likely to install cleanly and the full +50 path is reachable. -- The skill's design assumes baseline + latest both run cleanly; in practice latest-only with cap-at-84 is the workhorse path. The score-cap is doing real work, not just a fallback. - -**Keep-box-on-inconclusive.** When `verify-inconclusive` lands (Step 8c gave up, or Step 9 score < 60), **skip the cleanup trap** for this run if the box was provisioned by this run — set `PROVISIONED_NEW=0` before the trap fires so the EXIT handler is a no-op. Print the `brev shell "$INSTANCE_NAME"` command and an explicit `brev delete "$INSTANCE_NAME"` reminder in the run output so the maintainer can triage and clean up manually. Reused boxes stay regardless. Ship-failed verifications are the exact case where having an inspectable artifact pays for itself; an unbounded sleep-and-delete in the background isn't reliable across session ends, so we leave deletion explicit. - ---- - -## Step 12: Log to Activity - -After each issue (verified, inconclusive, by-design, or infra-failed), append to `${VERIFY_STALE_LOG_DIR:-$HOME/development/daily-rhythm/activity}/nemoclaw-verify-stale-log.md`. The default path matches the personal-organizer convention; export `VERIFY_STALE_LOG_DIR` to point elsewhere (CI, shared volume, etc.). Create the directory if missing — do not assume it exists. - -```markdown -### NVIDIA/NemoClaw#<number> — <title> -**Date:** YYYY-MM-DD -**Reported on:** v0.0.31 -**Verified on:** v0.0.34 -**Environment:** CPU | GPU (<instance type>) -**Box:** reused <name> | provisioned <name> | local (no Brev — Step 6.7 short-circuit) -**Baseline install:** succeeded | failed (degraded mode) -**Baseline match:** validated (verbatim) | validated (synth) | failed (verify-inconclusive) | skipped -**Latest install:** succeeded | failed (infra error) -**Latest result:** not-reproduced (clean) | still-reproduces | partial / flake | n/a (skipped 8d) -**Confidence:** 88 / 100 | n/a (still-reproduces) -**Label applied:** fixed-on-latest | verify-inconclusive | status: wont-fix | none (still-reproduces) | none (infra) -**Brev wall time (approx):** N min - ---- -``` - -Create the file if missing, with this header: - -```markdown -# NemoClaw — Verify Stale Log - -A running record of stale-issue verification runs on NVIDIA/NemoClaw. -Persisted via daily-rhythm to GitLab. - ---- -``` - -At end of a batch session, prepend a session summary: - -```markdown -## YYYY-MM-DD — Verify Session -**Issues considered:** N -**Verified `fixed-on-latest`:** N -**Marked `status: wont-fix` (by-design path):** N -**Marked `verify-inconclusive`:** N -**Local-first short-circuits (no Brev cost):** N -**Skipped (Windows / macOS / integration / no version):** N -**Infra failures:** N -**Brev wall time:** N min · approx $X.XX - ---- -``` - -Never stage or commit the log to the NemoClaw repo. - ---- - -## Cadence - -- **Weekly cron** — Monday morning, batch mode, ≤15 issues (the Step 1 cap, sliced after Step 3/4 filters). -- **Manual** — invoke with a single issue number anytime. - ---- - -## Out of Scope (v1) - -- Auto-closing issues. Always tag-only; a human pulls the trigger. -- macOS verification *via the Brev path*. Brev offers no macOS instances. The Step 6.7 local-first short-circuit *does* run on a maintainer's macOS laptop — so manual single-issue runs against pure-CLI bugs work on macOS. The weekly batch cron is Linux-only because that path always uses Brev. -- Issues requiring third-party integration credentials (Slack, Discord, Telegram, Hermes, OpenClaw, WeChat). -- Service-account bot identity. v1 runs under each maintainer's own GitHub credentials. -- Versioned labels. A single `fixed-on-latest` label is swept on each release cut. - ---- +## Steps 7–12 — Execution, Scoring, and Comment -## Companion Behavior +Once a candidate has cleared Step 6.7's local-first short-circuit and a Brev run is committed to, the rest of the workflow lives in **[reference/execution-and-comment.md](reference/execution-and-comment.md)**: -`nemoclaw-maintainer-cut-release-tag` sweeps `fixed-on-latest` and `verify-inconclusive` from all open issues at release time. Without that sweep, "latest" drifts and verifications go stale silently. The by-design path uses the existing repo `status: wont-fix` label; that label is **not** swept (it's also applied for non-skill reasons such as scope or priority decisions, and clearing it would erase human triage work). +- **Step 7** — Reuse or provision the Brev box (concurrency cap, runtime SKU pick, file-based API key copy). +- **Step 8** — Validate the reproducer on baseline, comprehensive reset, install latest, run again. Sub-steps cover dependency bootstrap, brev-exec quirks, synth-repro retry, architectural-drift check, performance and rebuild-cycle bug classes. +- **Step 8.5** — Detect "behavior changed by design" (three signals; short-circuits Brev cost on intentional removals). +- **Step 9** — Score confidence (+50 / +25 / +25 / −30 / −50; cap-at-84 when baseline didn't validate). +- **Step 10** — Compose and post the comment (redaction, 300-word ceiling, three templates, unanswered-question variant when Step 3 sets `UNANSWERED_MAINT_LOGIN`). +- **Step 11** — Infra failure handling (sandbox-build rot is the dominant failure for any version >5–7 patches behind). +- **Step 12** — Log to the activity file. +- **Cadence**, **Out of Scope (v1)**, and the **Companion Behavior** note (release-tag sweep) live there as well. diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md new file mode 100644 index 00000000000..3c775c496e0 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md @@ -0,0 +1,1167 @@ +<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> +<!-- SPDX-License-Identifier: Apache-2.0 --> + +# verify-stale — Execution, Scoring, and Comment Reference + +This file holds Steps 7–12 of the `nemoclaw-maintainer-verify-stale` workflow (everything after the candidate has cleared the local-first short-circuit in Step 6.7). The parent SKILL.md handles candidate filtering, version parsing, environment classification, reproducer extraction, preconditions, and the local-first decision; once a Brev run is committed to, follow this file. + +## Contents + +- **[Step 7: Reuse or Provision a Brev Box](#step-7-reuse-or-provision-a-brev-box)** — concurrency cap, runtime CPU SKU picking, file-based API key copy, cleanup trap. +- **[Step 8: Validate on Baseline, Verify on Latest](#step-8-validate-on-baseline-verify-on-latest)** — comprehensive reset, baseline install (Step 8a), reproducer dependency bootstrap (8a.5), brev-exec environment quirks (8a.5b), baseline run (8b), synth-repro retry (8c), latest install (8d), architectural-drift check (8d.5), performance-bug verification (8e), rebuild-cycle verification (8f). +- **[Step 8.5: Detect "Behavior Changed by Design"](#step-85-detect-behavior-changed-by-design)** — three signals, related-failure-mode pre-check, test-coverage check, self-verification, by-design comment template. +- **[Step 9: Score Confidence](#step-9-score-confidence)** — +50 / +25 / +25 / −30 / −50 rubric, baseline-validation cap-at-84, path extraction (commits-touched), PR-search (PR-mention). +- **[Step 10: Compose and Post the Comment](#step-10-compose-and-post-the-comment)** — redaction table (HTML→text pre-pass; JWT/PAT/NVAPI/base64/internal-host/email patterns), comment-authoring principle (300-word ceiling), per-verdict length defaults, mandatory caveats (cap, hardware substitution, verification mode, link self-verify), three templates (fixed/inconclusive, still-reproduces, by-design), unanswered-question prefix and dual @-mention variant. +- **[Step 11: Infra Failure Handling](#step-11-infra-failure-handling)** — sandbox-build rot is the dominant failure mode for any version >5–7 patches behind; cap-at-84 with reporter @-mention is by design. +- **[Step 12: Log to Activity](#step-12-log-to-activity)** — per-issue and per-session entries to `~/development/daily-rhythm/activity/nemoclaw-verify-stale-log.md`. +- **[Cadence](#cadence)** — weekly cron + manual single-issue invocation. +- **[Out of Scope (v1)](#out-of-scope-v1)** — auto-close, macOS verification, integration-credential bugs, service-account bot, versioned labels. +- **[Companion Behavior](#companion-behavior)** — `nemoclaw-maintainer-cut-release-tag` sweeps verification labels at release time. + +--- + +## Step 7: Reuse or Provision a Brev Box + +The skill prefers reuse over provisioning. A pool of `verify-stale-*` boxes (CPU and GPU) can be kept warm; reuse the matching one if available, otherwise provision. + +```bash +# Auth + install URL already verified by Step 6.5 — no need to re-check or auto-login here. + +# Determine class from Step 5: "cpu" or "gpu" +INSTANCE_CLASS="cpu" # or "gpu" + +INSTANCES=$(brev ls --json) + +# Look for an existing running verify-stale-* box matching the required class. +# CPU boxes have no .gpu field set; GPU boxes do. +EXISTING=$(echo "$INSTANCES" | jq -r --arg class "$INSTANCE_CLASS" ' + .[]? + | select(.name | startswith("verify-stale-")) + | select(.status == "RUNNING") + | select(($class == "gpu" and (.gpu // "" != "")) + or ($class == "cpu" and (.gpu // "" == ""))) + | .name' | head -1) + +PROVISIONED_NEW=0 + +if [ -n "$EXISTING" ]; then + INSTANCE_NAME="$EXISTING" + echo "Reusing existing verification box: $INSTANCE_NAME" +else + # Concurrency cap: refuse if 4+ verify-stale-* boxes are already running. + # Filter on .status to match the reuse query above — counting non-running boxes + # would falsely block provisioning when prior boxes are stopped but not deleted. + RUNNING=$(echo "$INSTANCES" | jq '[.[]? | select(.name | startswith("verify-stale-")) | select(.status == "RUNNING")] | length') + if [ "$RUNNING" -ge 4 ]; then + echo "ERROR: 4 verify-stale boxes already running. Wait for one to finish or reuse." + exit 1 + fi + + INSTANCE_NAME="verify-stale-${ISSUE_NUMBER}-$(date +%s)" + + if [ "$INSTANCE_CLASS" = "gpu" ]; then + # brev create auto-selects the cheapest GPU meeting the defaults + # (>=20GB VRAM, >=500GB disk, compute >=8.0). Override with --type if needed. + brev create "$INSTANCE_NAME" + else + # CPU case: pick the cheapest stoppable Linux SKU at runtime so the skill doesn't rot when + # SKUs change. Bias the floor by reproducer-implied memory needs — the cheapest 2 GB SKU + # cannot load a 4.8 GiB Ollama probe, and onboard fails at provider validation before any + # sandbox-creation code runs. Surfaced during the #2007 e2e run (wasted ~25 min on a 2 GB + # box that couldn't load `nemotron-3-nano:4b`). + # + # Memory floor heuristic: + # - Reproducer references Ollama or vLLM or names a model tag (e.g. `nemotron-3-nano:4b`, + # `llama3:8b`) -> floor 16 GB (covers ~5 GB model + sandbox + gateway overhead). + # - Reproducer touches sandbox onboarding without a local model server -> floor 8 GB. + # - Pure CLI-surface bug (no sandbox, no model) -> floor 4 GB. + # Override the auto-pick by exporting VERIFY_STALE_CPU_TYPE if the team has hard preferences. + CPU_RAM_FLOOR=${CPU_RAM_FLOOR:-8} + CPU_TYPE=${VERIFY_STALE_CPU_TYPE:-$(brev search cpu --sort price --json \ + | jq -r --argjson floor "$CPU_RAM_FLOOR" \ + '[.[] | select(.stoppable == true and .ram_gb >= $floor)] | .[0].type')} + [ -n "$CPU_TYPE" ] || { echo "ERROR: no stoppable CPU SKU with >= ${CPU_RAM_FLOOR} GB RAM"; exit 1; } + brev create "$INSTANCE_NAME" --type "$CPU_TYPE" + fi + + PROVISIONED_NEW=1 +fi + +# Cleanup runs on success, error, and SIGINT. +# Delete only what we provisioned. Reused boxes stay warm for next time. +# `brev delete` is non-interactive by default — there is no --yes flag, and passing one errors. +echo ">>> Brev instance: $INSTANCE_NAME (provisioned_new=$PROVISIONED_NEW; manual cleanup: brev delete $INSTANCE_NAME)" +trap '[ "$PROVISIONED_NEW" = "1" ] && brev delete "$INSTANCE_NAME" >/dev/null 2>&1 || true' EXIT +``` + +Wallclock cap per verification: **60 minutes** default. The cap accommodates two full install passes (baseline + latest), comprehensive resets between them, and any reproducer dependency bootstrapping (Step 8a.5) — most of which run sequentially against a single Brev box. Bugs that genuinely require more than an hour to manifest fall out of v1 scope; if a provisioned box isn't ready in time, abort and treat as an infra failure (Step 11). + +The previous design had a 25-min default with a 60-min extension for time-sensitive bugs (`memory leak`, `over time`, etc.). That split optimised for the wrong constraint — most issues fit comfortably under 60 min, and the keyword-based extension forced re-runs whenever a real install or bootstrap took longer than the optimistic 25-min budget. Single 60-min cap removes that paper cut. + +--- + +## Step 8: Validate on Baseline, Verify on Latest + +Two-pass design. + +- **Baseline pass (8a–8c):** install the **reported version**, run the reproducer, confirm it actually exposes the bug as described. This is the gate that proves the script is real. +- **Latest pass (8d):** install **latest**, run the validated reproducer. This is what the confidence score is built on. + +Without the baseline gate, a clean run on latest is ambiguous: maybe the bug really got fixed, maybe the script was never capable of triggering it. The baseline disambiguates. + +### Comprehensive reset (run before each install) + +NemoClaw spawns OpenShell sandboxes (containers), runtime services, and listening processes. A naive `rm -rf ~/.nemoclaw` doesn't clean those — the latest install would inherit baseline state and contaminate the result. Use this fuller reset between installs: + +```bash +RESET=$(cat <<'SCRIPT' +nemoclaw destroy --all --force 2>/dev/null || true +# Anchor pkill patterns to "/nemoclaw" / "/openshell" path components so the kill doesn't +# match unrelated processes that happen to mention these strings (including the agent +# harness running this skill if its working dir contains the word). +pkill -9 -f '/nemoclaw([[:space:]]|$)' 2>/dev/null || true +pkill -9 -f '/openshell([[:space:]]|$)' 2>/dev/null || true +docker ps -a --filter "name=openshell-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true +docker ps -a --filter "name=nemoclaw-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true +# Sandbox state lives in ~/.openclaw (default-writable since #2227); ~/.nemoclaw holds CLI state. +# Wipe both so the latest install starts clean. +rm -rf ~/.nemoclaw ~/.openclaw 2>/dev/null +sudo -n rm -f /usr/local/bin/nemoclaw 2>/dev/null || true +sudo -n rm -rf /usr/local/lib/nemoclaw 2>/dev/null || true +for port in 8080 18789 9119; do fuser -k -n tcp $port 2>/dev/null || true; done +true +SCRIPT +) +``` + +Idempotent — fails silently when there's nothing to clean. Run via `brev exec "$INSTANCE_NAME" "$RESET"` before 8a's install and again before 8d's install. + +**Sudo precondition.** All `sudo` invocations use `sudo -n` (non-interactive) so they fail fast instead of hanging on a password prompt. The skill assumes the Brev image's default user has passwordless sudo configured — Brev's stock images do; custom images may not. If `sudo -n` fails, the binary cleanup is best-effort and a stale `/usr/local/bin/nemoclaw` may persist. The user-local install path (`~/.nemoclaw`) is fully reset regardless. + +### Step 8a: Install reported version + +The installer accepts the target ref via the `NEMOCLAW_INSTALL_TAG` env var (verified against `install.sh` source — defaults to `latest` if unset). It is **not** a `--version` flag. + +```bash +brev exec "$INSTANCE_NAME" "$RESET" + +# Pass the provider env vars through so install.sh's bundled `[3/3] Onboarding` step +# doesn't fall back to the default `build` (NIM) provider — which requires NVIDIA_API_KEY +# and otherwise fails the install with a misleading error. When NEMOCLAW_PROVIDER=ollama +# (the common case), the bundled onboard uses the local Ollama we set up in Step 8a.5 +# and either succeeds (ideal) or fails on a real Dockerfile/sandbox-build issue (which +# is what we want to detect). Pass NVIDIA_API_KEY only if the maintainer provided one +# at Step 5's prompt. +# Read NVIDIA_API_KEY from ~/.nvidia-api-key on the BOX (not from this shell's argv). +# The Step 5 propagation block already brev-copy'd the key file with 600 perms. +brev exec "$INSTANCE_NAME" " + if [ -f ~/.nvidia-api-key ]; then export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key); fi + NEMOCLAW_INSTALL_TAG=$REPORTED_VERSION \ + NEMOCLAW_NON_INTERACTIVE=1 \ + NEMOCLAW_PROVIDER=${NEMOCLAW_PROVIDER:-ollama} \ + NEMOCLAW_MODEL=${NEMOCLAW_MODEL:-nemotron-3-nano:4b} \ + NEMOCLAW_SANDBOX_NAME=verify-stale-install \ + bash -c 'curl -fsSL $INSTALL_URL | bash' +" || BASELINE_INSTALL_FAILED=1 + +# Verify the resolved install version matches the requested version. This guards against the +# `VAR=val curl ... | bash` shell-scoping footgun where the env var binds to curl, not the +# downstream bash, and the install silently falls through to "latest". Surfaced during a +# rot-debugging investigation where v0.0.36 was silently installed when v0.0.26 was requested +# and several minutes of "convincing" output ran before anyone noticed. Always print the +# resolved state, never trust the requested state. +RESOLVED=$(brev exec "$INSTANCE_NAME" "bash -lc 'nemoclaw --version'" 2>&1 | tail -1) +echo "[verify-stale] baseline requested: $REPORTED_VERSION; resolved: $RESOLVED" +case "$RESOLVED" in + *"$REPORTED_VERSION"*) ;; # match — proceed + *) + echo "ERROR: baseline install resolved to '$RESOLVED' but $REPORTED_VERSION was requested." + echo " Common cause: env-var scoping in the install command. Verify the env vars are on" + echo " the BASH side of the curl|bash pipe, not the curl side. Setting" + echo " BASELINE_INSTALL_FAILED=1 to prevent verifying against the wrong version." + BASELINE_INSTALL_FAILED=1 + ;; +esac + +# The bundled onboard creates a sandbox name we don't want carrying through to the reproducer. +# Use a hyphen-only name (NemoClaw's name validator rejects underscores). Destroy it so the +# reproducer starts from a clean state. +brev exec "$INSTANCE_NAME" "sg docker -c 'nemoclaw destroy --all --force 2>/dev/null || true'" +``` + +If install fails (old releases rot — installer URLs, deps, OS images all drift over time, or the in-image Dockerfile patch step asserts against a code shape that's since changed), set `BASELINE_INSTALL_FAILED=1` and **skip 8b/8c**, going straight to 8d. Note "baseline-install-skipped" or "baseline-build-skipped" in the final comment depending on which phase rotted. Step 9's scoring rule handles the degraded mode (cap at 84). + +**The reproducer's own `nemoclaw onboard` (Step 8b) must pass `--fresh`.** If install.sh's bundled onboard was in an in-progress or failed state when we destroyed the install sandbox, the reproducer's onboard would error with `Previous onboarding session failed. Re-run with --fresh to discard it`. `--fresh` ensures a clean start. + +### Step 8a.5: Bootstrap reproducer dependencies + +Brev's stock CPU images ship with NemoClaw installable but not the broader ecosystem the reproducer may need — local model servers (Ollama, vLLM), inference providers, third-party CLIs. **Default to maximum faithfulness: install the actual dependency the reporter used rather than substituting a stub.** Substituting trades faithfulness for speed; that trade is rarely worth it on a 60-min budget, and it almost always introduces a confound that makes the verdict less trustworthy. + +**When to bootstrap (not substitute):** + +- The reproducer references a specific model/server runtime (`NEMOCLAW_PROVIDER=ollama`, `NEMOCLAW_PROVIDER=vllm`, etc.). +- The reproducer references a specific model name with a tag (`nemotron-3-nano:4b`, `llama3:8b`, etc.). +- The reporter's environment in the issue body shows a configured provider (e.g., `OpenShell CLI: 0.0.26` plus an Ollama running on host). + +**When to substitute (with -30 penalty):** + +- Provider requires an API key the skill cannot safely supply (NIM, OpenAI, Anthropic, etc.). Stubbing a key won't pass validation faithfully and a real key shouldn't sit in a verify-stale run. Apply the -30 penalty (treat as synth-repro per Step 8b) and document the substitution in the comment. +- The bug is *provably* independent of the dependency (e.g., a CLI argument-parsing bug that errors before any provider runs). Note this explicitly in the comment. + +**Canonical bootstraps:** + +```bash +# Ollama + a specific model. +# The Ollama installer registers a systemd service (`ollama.service`) so the +# daemon survives between brev exec calls. +brev exec "$INSTANCE_NAME" "curl -fsSL https://ollama.com/install.sh | sh" +brev exec "$INSTANCE_NAME" "sudo systemctl start ollama && sleep 3" +brev exec "$INSTANCE_NAME" "ollama pull <model>" +brev exec "$INSTANCE_NAME" "ollama list" # confirm before continuing +``` + +```bash +# vLLM + a model (HuggingFace-hosted). +brev exec "$INSTANCE_NAME" "pip install --quiet vllm" +brev exec "$INSTANCE_NAME" "nohup python -m vllm.entrypoints.openai.api_server --model <model> --host 127.0.0.1 --port 8000 >/var/log/vllm.log 2>&1 &" +brev exec "$INSTANCE_NAME" "sleep 30 && curl -fsS http://127.0.0.1:8000/v1/models" +``` + +Bootstrap **once before Step 8b's baseline run** and reuse for Step 8d's latest run. Don't reset Ollama/vLLM state between baseline and latest in the comprehensive reset — model downloads are expensive and unrelated to the NemoClaw install. Adjust the reset script to skip these external services explicitly if needed. + +**If bootstrap fails** (network issue pulling the model, service won't start, etc.), this is an infra failure — abort to Step 11. Do not silently substitute; the user opted into faithfulness for a reason. + +**Ollama coverage table.** Ollama is the default provider for verification runs because it's free, local, and self-hosted. It covers most bug classes faithfully but not all. Use this table to decide whether Ollama is sufficient or whether Step 5's API-key prompt should fire: + +| Bug class | Ollama covers? | Notes | +|---|---|---| +| CLI surface (subcommand parsing, flag handling, oclif dispatch) | ✓ Always | Provider not exercised | +| Sandbox structure (build, file permissions, mounts, layout) | ✓ Always | Provider not exercised | +| Networking / policy (port forwards, NAT, egress rules, channels guards) | ✓ Always | Provider not exercised | +| Generic inference flow (does an agent turn complete, does the proxy route correctly) | ✓ Usually | Ollama can fail in the same shape as NIM/Gemini for most flow bugs | +| Provider-specific behavior (`Provider: NVIDIA` symptom, NIM-only error handling, `Provider: Gemini` quirks) | ✗ No | Different code paths; substitution doesn't exercise the bug | +| Model-specific behavior (`gemini-flash-3-preview` doesn't handle prompt X, `nemotron-3-nano:4b` works fine) | ✗ No | Wrong model = wrong outputs | +| Ollama-shape-specific (#2519 "Ollama-local 401" — local-vs-networked Ollama config) | △ Sometimes | A generic Ollama install may or may not reproduce; may need specific configuration | +| Performance / latency on specific silicon | ✗ No | Hardware substitution caveat (Step 10) and Step 8e perf rubric apply | +| Quota / rate-limit / API-key validation | ✗ No | Ollama doesn't have those failure modes | + +When the table says ✗ No or △ Sometimes, Step 5's API-key prompt fires. When it says ✓, proceed with Ollama and skip the prompt. + +### Step 8a.5b: Brev exec environment quirks + +Two non-obvious gotchas surfaced during the #2007 e2e run that every subsequent `brev exec` call has to handle. Encode them once here so reproducer scripts don't have to relearn each time. + +**PATH does not include `~/.local/bin` in non-login shells.** `nemoclaw`'s installer drops a shim at `~/.local/bin/nemoclaw` and updates PATH via `~/.bashrc` / `~/.profile`. `brev exec` spawns non-login, non-interactive shells that don't source those files, so a bare `brev exec "$INSTANCE" "nemoclaw --version"` returns `command not found` on a freshly-installed box. Fix: every reproducer script must explicitly export PATH at the top, OR every `brev exec` call must wrap with `bash -lc '...'`. + +```bash +# Reproducer scripts: prepend this line. +export PATH="$HOME/.local/bin:$PATH" + +# Or equivalently when calling brev exec ad-hoc: +brev exec "$INSTANCE" "bash -lc 'nemoclaw --version'" +``` + +**Docker group requires `sg docker -c '...'` after `usermod -aG`.** Adding the user to the `docker` group (`sudo usermod -aG docker ubuntu`) takes effect for new login sessions, but `brev exec` calls in the same Brev session keep the old gid. The reproducer's `nemoclaw onboard` will fail with `permission denied while connecting to /var/run/docker.sock` unless the call runs in a subshell with the docker group active. + +```bash +# Reproducer execution: wrap with sg docker. +brev exec "$INSTANCE" "sg docker -c 'bash ~/reproducer.sh'" +``` + +Both patterns appear in the canonical setup script committed alongside the skill (or are encoded in your reproducer wrapper). Don't rely on the user discovering them mid-run. + +**`openshell sandbox exec` argument-order footgun.** When the reproducer needs to run a command *inside* the sandbox (channels-guard checks, in-sandbox file inspection, etc.), the correct non-interactive form uses `-n <name>` and a `--` separator: + +```bash +# Correct: +openshell sandbox exec -n ai -- bash -c 'source /sandbox/.bashrc; openclaw channels add telegram; echo "EXIT=$?"' + +# Wrong (silently auto-detects sandbox by "last used", stuffs the leftover positional +# `ai` into bash's $0, prints "/bin/bash: line 1: ai: command not found" — the +# reproducer appears to fail but actually never ran inside the sandbox at all): +openshell sandbox exec ai bash -c '...' +``` + +Issue #2592's first run hit this — wasted ~15 min before the maintainer noticed. Always use the `-n <name> -- <cmd>` form when the reproducer touches in-sandbox commands. + +**`brev exec` SSH-drop re-execution guard.** Brev's CLI silently retries from the top when the SSH connection drops mid-run, producing two parallel reproducer executions (we hit this on #2592 — one onboard process clobbered another's state, and both got billed). Use a sentinel file in the reproducer wrapper to make the script idempotent: + +```bash +# At the top of the reproducer wrapper script: +SENTINEL=~/.verify-stale-running +if [ -f "$SENTINEL" ]; then + echo "ERROR: another verify-stale run is in progress (sentinel: $SENTINEL)." + echo " If you're sure no other run is active, rm $SENTINEL and re-invoke." + exit 1 +fi +trap 'rm -f "$SENTINEL"' EXIT +touch "$SENTINEL" +``` + +The sentinel survives an SSH drop because it lives on the Brev box's filesystem; the trap removes it on script exit. A second `brev exec` invocation that tries to retry from the top will hit the sentinel and bail instead of double-running. + +--- + +### Step 8b: Run reproducer on baseline, compare to issue symptom + +If `./reproducer.sh` exists (verbatim from Step 6), run it. Otherwise synth on demand from the issue body (apply −30 penalty now, locked in for the rest of the run). + +**Interactive subcommand handling.** Many `nemoclaw onboard` / `nemoclaw configure` invocations prompt for input and will hang in a non-interactive shell. Auto-detect such subcommands in the script and apply, in order: + +1. Add `--non-interactive` if the version supports it. +2. Add `--dangerously-skip-prompts` (issue #2168 confirmed this exists for at least some Jetson paths). +3. Pre-feed answers via stdin: `printf 'yes\n\n\n' | nemoclaw onboard ...` + +If none work, route the script to Step 8c (synth-repro) so the LLM can rewrite it using non-interactive equivalents. + +```bash +# `brev exec` spawns a non-login shell, so ~/.local/bin (where the nemoclaw binary lives +# after install) is not on PATH unless we export it. The reproducer script itself must +# use `sg docker -c '...'` blocks for any Docker-touching command — Step 8a.5b covers +# that requirement; double-wrapping with sg docker on the outer call breaks nested-quote +# escaping in some bash versions. +brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh +brev exec "$INSTANCE_NAME" 'export PATH="$HOME/.local/bin:$PATH" && bash ~/reproducer.sh' 2>&1 | tee ./baseline-transcript.log +``` + +**Log-scraping (when `BUG_CLASS=log-only`).** Some bugs describe symptoms that show up in internal log files, not the reproducer's stdout/stderr — e.g., #1642 "see lots of error in openclaw log," #2611 "os.networkInterfaces guard errors." After running the reproducer, also pull the relevant logs from inside the sandbox and search them for the issue's symptom phrase: + +```bash +# Common NemoClaw / OpenClaw / OpenShell log paths inside the sandbox. +brev exec "$INSTANCE_NAME" "sg docker -c 'cat ~/.openclaw/logs/*.log /var/log/nemoclaw/*.log 2>/dev/null'" \ + | tee ./baseline-logs.log + +# Search the log capture for the issue's symptom phrase too, not just the transcript. +grep -F "<symptom phrase from issue body>" ./baseline-logs.log +``` + +For functional bugs the reproducer's stdout is sufficient; for log-only bugs the transcript may be clean but the log capture has the symptom. Both halves feed into the match rubric below. + +**Flake-detection retry.** Even for `functional` bugs, race-prone reproducers (TUI rendering, network policy negotiation, concurrent sandbox state) can produce inconsistent results. Run baseline three times if the first run shows the symptom inconsistently — same script, same env, just three back-to-back invocations. If the three runs disagree, that's signal: + +| 3-run baseline result | Verdict | +|---|---| +| All three reproduce the symptom | Strong baseline match → continue to 8d | +| All three are clean (no symptom) | Reproducer doesn't expose the bug on baseline → Step 8c synth-repro | +| Mixed (1 or 2 of 3 show the symptom) | Flake-prone reproducer. Note "flake suspected" in the comment; apply −25 to Step 9 score; downgrade `+50 latest clean` to `+25` because a clean latest run could just be the lucky path of an intermittent bug | + +Skip flake retry for `performance` and `rebuild-cycle` classes — those have their own multi-run rubrics in Steps 8e and 8f. + +**Match rubric.** LLM compares `baseline-transcript.log` to the issue's "Actual result" / error description. Match criteria, in order: + +1. **Exit code agrees** with what the issue describes (non-zero if issue describes a failure, zero if issue describes a wrong-output bug). Necessary but not sufficient. +2. **Symptom phrase match:** transcript contains a key error phrase from the issue (e.g., issue says `Permission denied on generate-openclaw-config.py`, transcript says `EACCES: permission denied, open '...generate-openclaw-config.py'` — semantic equivalence counts). +3. **Distinguish bug from infra noise:** generic network / DNS / auth errors don't count as a match unless the issue itself describes them. A bug about config parsing that fails at "could not resolve nvidia.com" is an infra failure, not a reproduction. + +**Fallback for issues without an explicit "Actual result" section.** Many bug reports describe a *behavioral* problem rather than a runtime error — e.g., "should default to a stable released version" (#1242), "configuration is not persisted across rebuilds" (#3030). These have no comparable error string. In that case: + +1. Use the issue's **full title + description** as the symptom signal. +2. Match if the reproducer's outcome **contradicts the issue's stated expected behavior** (or matches the stated wrong behavior). E.g., issue says "expected: stable release; actual: nightly", reproducer prints `nightly-build-2026.04.x` → that's a match. +3. If neither error string nor expected-behavior contradiction can be identified, route the script to Step 8c (synth-repro) — let the LLM produce a more diagnostic script that emits something testable. + +- **Match** → reproducer validated. Proceed to 8d. +- **No match** (silent pass, wrong error, infra noise, or no testable outcome): script has gaps. Proceed to 8c. + +### Step 8c: Synth-repro and retry on baseline + +LLM rewrites `./reproducer.sh` using the full issue context (description, environment, symptoms) **plus the baseline transcript** so it can react to what actually happened. Apply **−30 confidence penalty** (or keep it if 8b already applied it for the missing-verbatim case). + +```bash +brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh +brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcript-2.log +``` + +- **Match:** validated (with −30 baked in). Proceed to 8d. +- **Still no match:** mark `verify-inconclusive`. Post a comment that includes both reproducer attempts and both baseline transcripts with the message "couldn't establish a working reproducer for this bug on `$REPORTED_VERSION`." **Skip 8d** — there's nothing to verify on latest. + +### Step 8d: Install latest, run validated reproducer + +```bash +brev exec "$INSTANCE_NAME" "$RESET" +brev exec "$INSTANCE_NAME" " + if [ -f ~/.nvidia-api-key ]; then export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key); fi + curl -fsSL $INSTALL_URL | bash +" + +# Same resolved-version check as Step 8a — guard against env-var scoping or default fallthrough +# silently installing the wrong version. The latest install should resolve to $LATEST. +RESOLVED=$(brev exec "$INSTANCE_NAME" "bash -lc 'nemoclaw --version'" 2>&1 | tail -1) +echo "[verify-stale] latest requested: $LATEST; resolved: $RESOLVED" +case "$RESOLVED" in + *"$LATEST"*) ;; # match — proceed + *) echo "WARN: latest install resolved to '$RESOLVED' (expected match for $LATEST). Proceeding but flag in comment." ;; +esac + +# OpenShell version pin — surfaced from #1642's e2e run. Latest's blueprint.yaml may set +# `max_openshell_version` below what the OpenShell installer would otherwise grab. The +# baseline phase (Step 8a) installed whichever OpenShell was current at reported-version, +# which can be newer than latest's cap (e.g., reported v0.0.6 → installed openshell 0.0.37, +# latest v0.0.38 caps at 0.0.36, onboard preflight refuses to run). Re-pin from latest's +# repo so onboard preflight passes; if the new pin is OLDER than the installed binary, +# install-openshell.sh refuses the downgrade — fall back to direct GitHub download. +brev exec "$INSTANCE_NAME" ' + set -e + cd ~/NemoClaw + git fetch --depth 1 origin tag "'"$LATEST"'" 2>&1 | tail -2 + git checkout -- . 2>/dev/null || true + git checkout "'"$LATEST"'" 2>&1 | tail -2 + + MAX_OS=$(grep -E "^max_openshell_version:" nemoclaw-blueprint/blueprint.yaml 2>/dev/null | awk "{print \$2}" | tr -d "\"" | tr -d "v") + CUR_OS=$(openshell --version 2>&1 | grep -oE "[0-9]+\.[0-9]+\.[0-9]+" | head -1 || echo 0.0.0) + echo "[verify-stale] openshell pin: blueprint max=$MAX_OS, currently installed=$CUR_OS" + + if [ -n "$MAX_OS" ] && [ "$(printf "%s\n%s\n" "$CUR_OS" "$MAX_OS" | sort -V | tail -1)" != "$MAX_OS" ]; then + echo "[verify-stale] currently installed openshell ($CUR_OS) is newer than blueprint cap ($MAX_OS) — force-downgrading" + sudo rm -f /usr/local/bin/openshell + cd /tmp + curl -fsSL "https://github.com/NVIDIA/OpenShell/releases/download/v$MAX_OS/openshell-x86_64-unknown-linux-musl.tar.gz" -o openshell-pin.tar.gz + tar -xzf openshell-pin.tar.gz + sudo install -m 755 ./openshell /usr/local/bin/openshell + openshell --version + else + sudo bash scripts/install-openshell.sh 2>&1 | tail -3 + fi +' + +brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh +# Same PATH safeguard as the baseline call — non-login shells don't pick up ~/.local/bin +# automatically. The reproducer's internal `sg docker -c '...'` blocks cover Docker access. +brev exec "$INSTANCE_NAME" 'export PATH="$HOME/.local/bin:$PATH" && bash ~/reproducer.sh' 2>&1 | tee ./latest-transcript.log +``` + +If the install of **latest** fails (e.g. installer regression — see #3058 for a current example), this is an infra failure — see Step 11. Do not score or label the issue. + +If install succeeds, `latest-transcript.log` is the input to Step 9 scoring. + +For interactive debugging when something looks off: + +```bash +brev shell "$INSTANCE_NAME" +``` + +--- + +## Step 8d.5: Architectural-Drift Check + +Cross-version verification compares two moving targets: the reproducer assumes `$REPORTED_VERSION`'s tooling surface, and `$LATEST` may have rewritten the surface entirely. If the *tool* the reproducer relies on (CLI subcommand, output table, log file location) was reworked between the two tags, an "empty / clean output on latest" can mean either "bug fixed" OR "we're looking at a deprecated tracking surface." Without this check, the latter silently registers as the former — a class of false positive. + +**Detection** — pickaxe the diff between tags for the reproducer's tool name and watch for the CLI itself being touched, not just its consumers: + +```bash +# Extract the primary verification command from the reproducer (e.g. "openshell forward list"). +TOOL=$(grep -oE '\b(openshell|nemoclaw)[[:space:]]+[a-z-]+' reproducer.sh | sort -u) + +# Pickaxe each tool name across the version range. +for t in $TOOL; do + echo "=== drift check: $t ===" + git log "$REPORTED_VERSION".."$LATEST" -S"$t" --oneline -- src/ bin/ nemoclaw/src/ 2>&1 | head -5 +done +``` + +If a tool is touched, drift is suspected. + +**Multi-axis verification** — when drift is suspected, do not rely on the reproducer's expected output alone. Pick OS-level surfaces that would show the buggy state regardless of which CLI tracks it. For port-forwarding bugs (the #2007 case), the canonical five-axis pattern: + +| # | Surface | Command | +|---|---|---| +| 1 | Reproducer's stated check | as written in the issue body | +| 2 | Host TCP listeners | `sudo ss -tlnp` | +| 3 | iptables NAT redirects | `sudo iptables -t nat -L -n` | +| 4 | Docker port mappings | `docker ps --format '{{.Names}} {{.Ports}}'` | +| 5 | Active SSH tunnels | `ps -ef \| grep 'ssh.*-L'` | + +Adapt the axes to the bug class. For filesystem bugs: `find`, `lsattr`, `stat`. For network policy bugs: `iptables -L`, container netns, gateway logs. The principle is the same — pick at least three independent surfaces that would each independently show the buggy state if it were present. + +**Action when drift is suspected:** + +- Run the multi-axis pattern after Step 8d's reproducer. +- The verdict requires **every relevant axis to be clean** — not just the reproducer's surface — before claiming `fixed-on-latest`. +- Quote the multi-axis evidence in the Step 10 comment as a table; this is exactly what makes "fixed" defensible when the original tooling no longer reflects the underlying behavior. +- If any axis still shows the buggy state, the bug is NOT fixed even if the reproducer's surface is clean. Escalate to "still reproduces" (Step 9 special case). + +**When drift is NOT suspected** (the reproducer's tool is unchanged in the version range): the reproducer's expected output is sufficient, no multi-axis verification needed. + +--- + +## Step 8e: Performance-Bug Verification (when `BUG_CLASS=performance`) + +Performance bugs (#2598 "10s P50", #2600 "hangs ~2 min", #2733 Ollama tool-call leak over time) can't be answered by the standard exit-code + symptom-phrase rubric — one clean reproducer run doesn't tell you the p50 budget is met; one slow run doesn't tell you the bug still reproduces. Replace Step 8b's match with a measurement-and-distribution rubric: + +1. **Parse the SLA from the issue body.** Extract numeric latency thresholds: `10s P50`, `200ms`, `under 5 seconds`, `~2 min`. Save as `SLA_P50_MS`, `SLA_P90_MS`, etc. If no numeric SLA is in the body, route to Step 8c synth-repro to ask the reporter (via comment) for one — without a target, the verdict is undefined. +2. **Run the reproducer N=10 times** on each side (baseline + latest), capturing per-run latency: + + ```bash + for i in $(seq 1 10); do + /usr/bin/time -f '%e' bash ~/reproducer.sh >/dev/null 2>>./latest-perf.log + done + ``` + +3. **Compute p50 and p90** for both sides. `sort -n ./latest-perf.log | awk 'NR==5'` for p50 of 10 runs. +4. **Match rubric:** + - Latest's p50 within the SLA AND baseline's p50 outside the SLA → bug fixed; same Step 9 scoring (subject to baseline-validation gate). + - Latest's p50 outside the SLA → bug still reproduces (Step 9 special case). + - Latest p50 within SLA AND baseline p50 also within SLA → reproducer doesn't actually exercise the bug; route to Step 8c synth-repro. + +**Hardware-substitution caveat.** Performance numbers are silicon-dependent. When the issue is `Platform: DGX Spark` or `Platform: GB10` and we're measuring on a Brev x86 GPU SKU, the comment must say so explicitly: a Brev p50 of 1.5s on a `H100` does not prove the DGX Spark p50 is fixed. Cap the score at 60 unless the bug is clearly silicon-independent (e.g. an algorithmic regression in user-space JS that would manifest the same on any silicon). + +--- + +## Step 8f: Rebuild-Cycle Verification (when `BUG_CLASS=rebuild-cycle`) + +Rebuild-cycle bugs (#2701 "Pod recreate wipes `/tmp/nemoclaw-proxy-env.sh`," issues describing "configuration is not persisted across rebuilds") only manifest when sandbox state crosses a destroy/recreate boundary. A single onboard run can't trigger the symptom. Replace Step 8b's match with a run-rebuild-rerun harness: + +1. **First onboard.** Run the reproducer once to establish initial state. Capture relevant artifacts (config files, env vars, sandbox metadata) — the issue body usually names what should persist: + + ```bash + brev exec "$INSTANCE_NAME" "sg docker -c 'cat <files-mentioned-in-issue> 2>&1'" | tee ./pre-rebuild.log + ``` + +2. **Trigger the rebuild.** Use `nemoclaw destroy --all --force` followed by `nemoclaw onboard` with the same env vars. Do NOT comprehensive-reset between (the point is to test the destroy/recreate, not start from scratch). + +3. **Re-capture the same artifacts** post-rebuild: + + ```bash + brev exec "$INSTANCE_NAME" "sg docker -c 'cat <same-files> 2>&1'" | tee ./post-rebuild.log + ``` + +4. **Diff and match.** The bug is "X gets wiped / changes / regresses across rebuild." Compare pre-rebuild vs post-rebuild captures to the issue's expected behavior: + - Pre and post agree (artifact preserved) AND issue says it should be preserved → bug fixed + - Pre and post differ (artifact wiped) AND issue says it gets wiped → bug still reproduces + - Pre and post agree AND issue says it gets wiped → reproducer doesn't exercise the bug; Step 8c synth-repro + +The harness still uses Step 9's scoring framework — `+50 latest clean (artifact preserved)`, etc. — but the "what gets compared" axis is the diff, not the symptom phrase. + +--- + +## Step 8.5: Detect "Behavior Changed by Design" + +Before scoring, check whether the symptom is intentional. Some bugs are filed against behavior that was **deliberately changed or removed** in a merged PR — running the standard rubric on these produces misleading verdicts. The symptom "still reproduces" but the right answer is "won't fix, see PR #X." Issue #2791 is the prototype: `config set` was removed in PR #2227, the reporter tested a version that already had it gone, and a standard rubric run would have buried that context under a low-confidence `verify-inconclusive` label. + +This step is split into substeps so the rigor is mechanical, not optional. Every claim in the final comment must be backed by a verifiable evidence block — a comment URL with quoted phrase, a commit SHA with diff range, or a grep command with its actual output. Hand-wavy claims fail Step 8.5d's self-verification pass and force a bail to `verify-inconclusive`. + +### Step 8.5a: Run signal detection + +Any single signal is sufficient to trigger the by-design branch. + +**Signal 1 — Maintainer attribution in comments.** Any comment by an author with `authorAssociation` of `MEMBER`, `OWNER`, or `COLLABORATOR` matches `removed in #\d+`, `removed in [Pp][Rr] ?#\d+`, `by design`, `wontfix`, `won't fix`, `not a bug`, or `intentional`. + +```bash +gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ + --jq '.comments[] + | select(.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR") + | select(.body | test("removed in #\\d+|by design|wontfix|won.t fix|not a bug|intentional"; "i")) + | {url, author: .author.login, body}' +``` + +Capture for evidence: comment URL + author login + the exact quoted phrase. + +**Signal 2 — Removal commit in range.** A commit between the reported version and `$LATEST` deletes the symbol implicated by the reproducer (CLI subcommand, function, flag). The commit subject does NOT need to mention "remove" / "delete" — many removals ride into a `refactor(...)` or `feat(...)` commit (e.g. PR #2227 removed `--dangerously-skip-permissions` under a `refactor(sandbox): ...` subject). Use git's pickaxe to find the responsible commit by content: + +```bash +# Pickaxe: list every commit whose diff changes the count of <symbol> occurrences. +# Reverse order so the earliest removal commit lands first in the list. +git log "$REPORTED_VERSION".."$LATEST" -S'<symbol>' --reverse --oneline -- src/ bin/ nemoclaw/src/ + +# Subject-keyword narrowing is only a SUPPLEMENTARY lookup — useful when the +# pickaxe returns many commits and you want to focus on the obviously-removal one. +git log "$REPORTED_VERSION".."$LATEST" --grep='remove\|delete\|drop\|deprecate' -i --oneline + +# For each candidate, confirm the diff actually deletes the symbol (not just renames or moves it). +git log -p <candidate-sha> -- src/ bin/ nemoclaw/src/ | grep -nE '^-.*\b<symbol>\b' +``` + +Capture for evidence: commit SHA + each `file:line` block of deletions touching the symbol. Note the commit's actual subject — don't assume it says "remove." + +**Signal 3 — Symbol absent in both reported version and latest.** The implicated symbol (e.g. `config set`) is not present in either tag's source tree — meaning the responsible change landed before the version the reporter tested. This is the #2791 case. + +```bash +git grep -n "<symbol>" "$REPORTED_VERSION" -- src/ bin/ nemoclaw/ # expect: zero matches (or shim-only — see sub-case) +git grep -n "<symbol>" "$LATEST" -- src/ bin/ nemoclaw/ # expect: zero matches (or shim-only) +``` + +Capture for evidence: both grep commands and their (empty) outputs. + +**Sub-case for signals 2 and 3 — vestigial deprecation shims.** It's common for a removed symbol to survive in latest *only* as a deprecation message (e.g., a CLI subcommand that prints `"--<flag> was removed; use <X> instead"` and exits non-zero). When a grep returns matches in latest, inspect each `file:line`. If every match is a deprecation stub with no functional effect on the bug-as-filed, signal 2 or 3 still fires; record the shim locations and behavior as a separate evidence block. Do not silently treat shims as functional code, and do not silently treat them as absence. + +### Step 8.5b: Pre-check related failure modes + +A by-design verdict says "the bug *as filed* can't reproduce." It does NOT say "every bug shaped like this is fixed." Before drafting the comment, search latest's source for code paths that could still produce the issue's described **symptom** (not the literal removed flag/symbol — the symptom). + +```bash +# Use the issue's symptom keywords, not the removed symbol. +git grep -nE "<symptom-keyword-1>|<symptom-keyword-2>" "$LATEST" -- src/ nemoclaw/src/ +``` + +For #2168 the literal flag is `--dangerously-skip-permissions`, but the symptom is "sandbox created but not registered in CLI." Grepping for `register.*[Ss]andbox`, the readiness-gate / cleanup-failure path in `src/lib/onboard.ts` surfaces as a related-but-different way to produce an orphan sandbox. + +If a related failure mode is found, the by-design comment MUST include a "What's not literally the same bug" section that names it with `file:line`. Don't suppress the call-out by claiming "the symptom is impossible" when the symptom can be reached via a different path. + +### Step 8.5c: Check existing test coverage + +Search the repo for tests that exercise the NEW intended workflow (the one that replaced the removed symbol). Citing them strengthens the comment from "trust me, it was removed" to "the new workflow is exercised by these tests." + +```bash +git grep -lnE "<new-workflow-keyword>" -- test/ nemoclaw/src/ 2>/dev/null | head -5 +``` + +Cite at most three concrete test paths. If none exist, omit the section — do not invent paths. + +### Step 8.5d: Self-verification pass before posting + +Two passes, both required. + +**Evidence pass.** Re-run every grep / git / `gh` command cited in the evidence blocks. If any cited `file:line`, commit SHA, or quoted output doesn't reproduce on a fresh invocation, **stop and revise** — or bail to `verify-inconclusive` if the discrepancy can't be resolved. + +**Link pass.** Resolve at least one rendered markdown link from each section that has them — `What's structurally fixed`, `Vestigial references`, `Existing CI coverage`. Use `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 content if the path exists at the tag, 404 otherwise) or `curl -fsI <blob-url>` (returns 200 if the blob renders). A broken link is worse than no link — it suggests verification work that didn't actually happen. + +The cost of an incorrect "I checked and X is gone" claim in a public comment, or a 404 on a citation, is higher than spending a minute re-checking. This step exists because LLMs can confidently overstate and confidently invent paths; mechanical re-verification catches both. + +### Step 8.5e: If any signal fires + +- **Skip the Step 9 score table** entirely. The "exit 0 + expected output" axis doesn't apply when the expected output is no longer the contract. +- **Skip Brev provisioning** if the signal fires before Step 7 — a remote run would just confirm what static analysis already proved. (Signals 2 and 3 can run as soon as the reported version is parsed in Step 4.) +- **Apply label `status: wont-fix`** (the existing repo label — quote it on the CLI: `gh issue edit <num> --add-label "status: wont-fix"`). It's already in the Step 3 issue-type skip list, so a labelled issue is automatically excluded from future runs without needing a separate idempotency clause. +- **Use the by-design comment template below** instead of the standard Step 10 template. +- **@-mention the reporter** so they can object if the framing is wrong. +- **Never auto-close.** A maintainer pulls the trigger, same as the other label paths. + +### By-design comment template + +Mandatory sections in this order. Omit only the sections explicitly noted as omittable. + +**Tag-anchoring + linking rule.** Every `file:line` citation, commit SHA, and test-path reference in the rendered comment MUST be a clickable markdown link to the verified-on tag (e.g., `v0.0.35`), not the maintainer's working `HEAD`. Lines drift between tags and main; tag-anchored links keep the citations reproducible by anyone reading the comment months later. Bare paths force the reader to navigate manually — that's a usability bug, not a stylistic preference. + +Use these exact link formats: + +- File only: `[src/lib/onboard.ts](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/onboard.ts)` +- File:line: `[src/lib/onboard.ts:4965](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/onboard.ts#L4965)` +- File:line-range: `[src/lib/commands/sandbox/connect.ts:25-31](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/commands/sandbox/connect.ts#L25-L31)` +- Commit SHA: `[5956a61](https://github.com/NVIDIA/NemoClaw/commit/5956a612e18047b9ab85b3a7e89f6b5dedb29190)` — short SHA as the link text, full SHA in the URL +- Test file: `[test/e2e/test-double-onboard.sh](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/test/e2e/test-double-onboard.sh)` +- PR/issue references: bare `#NNNN` works — GitHub auto-links these in comments on the same repo, no manual URL needed. + +When greping for evidence, use `git grep -n "<symbol>" "$LATEST" -- ...` so the line numbers match the tagged blob. Then construct each link from `<file path> + verified-on tag + line number`. + +The Step 8.5d self-verification pass MUST resolve at least one rendered link (e.g., `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=v0.0.35` or a `curl -fsI` to the blob URL) and confirm it returns the expected file. A broken link defeats the purpose of including the citation. If any link fails to resolve, fix it or bail to `verify-inconclusive`. + +````markdown +## Stale-issue verification — behavior is by-design + +**Reported on:** v0.0.<X> +**Verified on:** v0.0.<Y> (PR #<NNNN> first shipped in v0.0.<Z>) +**Verification mode:** static analysis at the verified-on tag — no runtime reproduction. Step 8.5 by-design short-circuits Brev provisioning because the responsible code change is already proven by the diff between `$REPORTED_VERSION` and `$LATEST`. +**Outcome:** symptom reproduces against the reproducer as filed, but the implicated behavior was intentionally changed. + +### What's structurally fixed + +- `<file:line>` — `<one-sentence summary of the change at that location>` +- `<file:line>` — `<…>` + +The new workflow is `<one-sentence: how to do what the user was trying to do>`. + +### Vestigial references + +- `<file:line>` — `<deprecation behavior: e.g. "prints '--<flag> was removed; use <X> instead' and exits 1; no functional effect">` + +(Omit this section entirely when the symbol is fully gone with no surviving stubs.) + +### What's not literally the same bug + +`<one-sentence acknowledgement of the related failure mode found in Step 8.5b, with file:line>` — OR — `None. The symptom requires the removed symbol; no related code path produces it on latest.` + +### Existing CI coverage + +- `<test/path/file>` — `<one-sentence: what this test demonstrates about the new workflow>` + +(Omit when no direct test exists. Do not invent paths.) + +### Recommendation + +@<reporter> — please confirm the by-design framing is correct (the implicated `<symbol>` was intentionally removed, the original reproducer can no longer execute) and close as "won't fix / by design" if you agree. If a related symptom (e.g. `<related failure mode from above>`) is hitting you on ≥ v0.0.<Z>, please file a fresh issue with a v0.0.<Z>+ reproducer. + +`<NVBugs cross-ref line — see below>` + +<!-- nemoclaw-verify-stale v1 YYYY-MM-DD --> +```` + +**NVBugs cross-ref line.** If `NVBUGS_REF` was set in Step 4, append: + +> NVBugs<NVBUGS_REF without brackets> will need a separate update; closing this GitHub issue won't propagate. + +Otherwise omit the sentence. + +**If no signal fires:** continue to Step 9 normally. + +--- + +## Step 9: Score Confidence + +Start at 0. Apply each rule that fires. + +| Signal | Delta | +|---|---| +| Reproducer ran cleanly on **latest** (8d), exit 0, no bug symptom observed | +50 | +| Commits between reported version and `$LATEST` touch the implicated component (see "Path extraction" below) | +25 | +| A merged PR mentions this issue number or its symptom (see "PR search" below) | +25 | +| Reproducer was LLM-synthesized at any point (Step 8b synth or Step 8c retry) | −30 | +| Any partial error, warning, or flaky behavior in the latest run (8d) | −50 | + +Total is clamped to `[0, 100]`. + +### Path extraction (for the +25 commits signal) + +The skill needs to know *which* path to `git log v<reported>..$LATEST -- <path>` against. Apply in order, stop at the first that yields a non-empty path: + +1. **Stack trace / file path mentions in the issue body.** Grep the body for absolute paths under known install roots, then map to repo paths: + - `/usr/local/lib/nemoclaw/<rel>` → `<rel>` in repo (e.g., `scripts/generate-openclaw-config.py`) + - `/usr/local/bin/nemoclaw*` → `bin/` + - `~/.nemoclaw/<rel>` → most often runtime state, drop unless the bug is config-related → `src/lib/config/` + - In-repo paths (e.g., `bin/lib/policies.js` mentioned literally) → use as-is +2. **Component-label-to-directory map.** Pick the first match. Paths verified against the current repo layout — drop any path that doesn't exist on the tag at `$LATEST` rather than passing it to `git log`. + - `NemoClaw CLI` → `bin/`, `src/lib/`, `nemoclaw/src/commands/` + - `Sandbox` → `nemoclaw/src/blueprint/`, `nemoclaw-blueprint/` + - `OpenShell` → cross-repo (lives at `github.com/NVIDIA/OpenShell`, not in this repo). Skip the +25 signal for OpenShell-only issues; cross-repo `git log` is out of v1 scope. + - `Docker` → `Dockerfile`, `Dockerfile.base`, `scripts/install-openshell.sh`, `scripts/install.sh` + - `Getting Started` → `docs/`, `scripts/install.sh` + - `Integration: <X>` — no `src/lib/integrations/` exists in this repo. Skip the +25 signal for integration-component issues unless source 1 (file paths in body) yielded a path. +3. **Title keywords.** "policy" → `nemoclaw-blueprint/policies/`, `nemoclaw/src/blueprint/`. "inference" → `docs/inference/` is docs-only; skip the +25 signal unless source 1 surfaces actual code paths. + +If none of the above produces a path, **skip the +25 signal entirely** rather than guessing. Floating the +25 on every issue would inflate scores meaninglessly. + +### PR search (for the +25 PR signal) + +```bash +# Direct issue-number reference (covers most cases — "fixes #2861" etc.) +DIRECT_REF=$(gh pr list --repo NVIDIA/NemoClaw --state merged \ + --search "$ISSUE_NUMBER" \ + --json number,title,mergedAt,body \ + -q "[.[] | select((.body + \" \" + .title) | test(\"#$ISSUE_NUMBER\\\\b\"))]") + +# Symptom-phrase fallback (only if direct reference returns nothing) +if [ -z "$DIRECT_REF" ] || [ "$DIRECT_REF" = "[]" ]; then + SYMPTOM=$(extract first key error/symptom phrase from issue body, ~3-6 words) + SYMPTOM_REF=$(gh pr list --repo NVIDIA/NemoClaw --state merged \ + --search "\"$SYMPTOM\"" \ + --json number,title,mergedAt) +fi +``` + +Apply +25 if either query returns at least one PR with `mergedAt` strictly after the tag date of `$REPORTED_VERSION` (look up via `git log -1 --format=%cI v$REPORTED_VERSION`). PRs merged before the reporter even filed the issue can't have fixed it. + +If neither query returns anything, **skip the +25 signal**. + +**Baseline-validation gating.** The +50 weight assumes the reproducer was *validated* — i.e., it produced the bug symptom on baseline (Step 8b/8c match). If `BASELINE_INSTALL_FAILED=1` (Step 8a fall-through, baseline pass skipped — including the sandbox-build-rot case from Step 11), the +50 still applies but **cap the total at 84**. Corroboration signals (commits-touched-area, PR-mention) still raise the score within the cap but cannot lift it above 84. Without runtime baseline confirmation we don't have enough on our own to claim ≥85 — the cap forces the verdict into the 60–84 band where the reporter is asked to confirm. The previous draft of this rule had an "unless commits-touched OR PR-mention also fires" escape hatch that let inferred fix evidence bypass the cap entirely; that produced a misleading 100/100 on the #2007 e2e run despite zero baseline confirmation, and was tightened here. + +**Action (when latest run was clean — bug not reproduced):** + +| Score | Label | Comment | +|---|---|---| +| ≥85 | `fixed-on-latest` | Evidence-rich, no @-mention. | +| 60–84 | `fixed-on-latest` | Evidence-rich, **@-mention the original reporter** to confirm. | +| <60 | `verify-inconclusive` | Short, honest "couldn't verify" explanation. | + +**Special case: latest output matches the issue symptom (bug still reproduces on latest).** + +This is not a flake — the skill positively confirmed the bug is still live. Don't apply the +50 weight (the bug isn't fixed) and skip the score table entirely. + +- Post a "still reproduces on latest" comment with both transcripts. +- Apply **no label**. +- Include the marker `<!-- nemoclaw-verify-stale v1 YYYY-MM-DD -->` with today's date so the candidate filter applies the 7-day TTL (Step 3 idempotency). +- Next weekly run picks the issue back up after the TTL — if the bug gets fixed in the meantime, that run catches it. + +The skill **never closes issues** in any branch. A maintainer pulls that trigger after reviewing the label and comment. + +--- + +## Step 10: Compose and Post the Comment + +**Redaction pass before posting.** Run on **every** chunk of text quoted in the comment — issue body excerpts, baseline transcript, latest transcript, synth-repro scripts. Replace each match with `[REDACTED]`. The transcripts especially leak — they include full stdout/stderr from real installs and runs. + +**HTML → text pre-pass for issue body excerpts.** NV QA bodies are HTML; tokens nested in `<pre>` tags or HTML attributes (e.g. `<a href="https://user:tok@host/...">`) slip past the regex patterns below if the input still has tags. Convert to plain text first, then redact: + +```bash +TEXT=$(printf '%s' "$BODY_EXCERPT" | python3 -c ' +import html, re, sys +b = sys.stdin.read() +b = re.sub(r"<br\s*/?>", "\n", b) +b = re.sub(r"</?(p|div|tr|td|th|li|pre)[^>]*>", "\n", b) +b = re.sub(r"<[^>]+>", "", b) +print(html.unescape(b)) +') +# Now apply the regex table below to $TEXT. +``` + +Transcripts and synth-repro scripts are already plain text and skip the pre-pass. + +**Order matters and the patterns below are in execution order.** Longest, most-specific patterns first; generic catchalls last. Otherwise the catchall masks specific matches and you lose track of what was actually redacted (JWT vs session blob vs random base64). + +Patterns live in a fenced block (not a markdown table) because patterns 8 and 9 use regex alternation `|` — markdown tables would treat the literal `|` as a column delimiter, and escaping it as `\|` makes the regex match a literal pipe instead of an alternation, which silently breaks credential redaction. + +```regex +1. eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,} + → JWT tokens + +2. gh[pousr]_[A-Za-z0-9]{36,} + → GitHub PATs / install tokens + +3. (?i)nvapi-[A-Za-z0-9_-]{20,} + → NVIDIA API keys (NIM / build.nvidia.com) + +4. AKIA[0-9A-Z]{16} + → AWS access key IDs + +5. (?i)aws_secret_access_key\s*=\s*\S+ + → AWS secret keys + +6. (?i)authorization:\s*\S+ + → HTTP auth headers (often Bearer + JWT) + +7. URLs containing `@` before the host (e.g., https://user:pw@host/...) + → Basic-auth credentials in URLs + +8. (?i)(token|secret|password|api[_-]?key|bearer)[^\n]*[:=][^\n]* + → Inline credentials in env/config/log output + +9. \b\w+\.(nvidia\.internal|nv-internal\.com|nvidia\.dev)\b + → Internal hostnames (extend list per team) + +10. [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} + → Email addresses (PII) + +11. \b[A-Za-z0-9+/]{60,}={0,2}\b + → Long base64 blobs (likely keys/sessions; tune length to taste — too short hits legit data) +``` + +**File paths under the reporter's home directory** (`/Users/<name>/`, `/home/<name>/`) → replace with `~/`. Run last; catches incidental username PII. + +**Comment authoring principle.** Every section in a rendered comment must either change a reader's mind about the verdict, or be cut. Word counts follow from that — **300 is a hard ceiling** for the main verdicts (fixed-on-latest, wontfix). Simple cases (clear PR ref, deterministic check) land under 200. The principle generalizes: comments posted by this skill compete for a maintainer's attention against every other in-flight thread, and "AI-slop" prose — architectural sidebars, file:line citations the maintainer can find via the PR ref, bare-output reproductions when the load-bearing evidence is elsewhere, "if this verification is wrong, please reopen…" boilerplate — actively reduces the comment's signal-to-noise ratio. + +**For each section in a draft, ask: would the maintainer reach a different conclusion *without* this section? If no, delete.** Lessons accumulated from real runs: + +- **#2007 first draft (~750 words):** had a multi-paragraph "Architectural notes for QA reference" section that didn't change the verdict. Cut → 371 words. +- **#2604 first three drafts:** wavered between fixed-on-latest, still-reproduces, and by-design across iterations because each draft padded the verdict with prose that didn't ground it. Final 190-word draft cut a maintainer-note sidebar about platform attribution, a bare-status output reproduction, and a file:line citation of the source — none affected the verdict, all were AI-slop padding. Rule learned: **before drafting any prose, name the verdict in one sentence; if a section doesn't directly support that one sentence, cut it before writing it.** + +**Per-verdict length defaults:** + +| Verdict | Target | Rationale | +|---|---|---| +| `fixed-on-latest` | **200–300 words** | Header + evidence + verdict + @-mention. Add hardware-substitution caveat or related-failure-mode section only if they shift the maintainer's read. If you're past 300, you're padding. | +| `wontfix` (by-design) | **200–300 words** | Structurally-fixed + vestigial + what's-not-the-same-bug, each one to two sentences max. The PR ref carries the detail; the comment carries the verdict. | +| `verify-inconclusive` | 100–200 words | One paragraph naming what the skill couldn't establish. No transcripts beyond a single quoted line. | +| **Still-reproduces (no label)** | **30–80 words** | The reporter already has the symptom; the maintainer can see the issue is open. The skill is just confirming + setting the TTL marker. **No transcripts** (the issue body has them), **no @-mention** (the reporter knows their bug is real), **no architectural prose**. One sentence stating "skill ran reproducer on `<latest>`, symptom still present" + one sentence on any partial-fix PR if relevant + marker. That's it. | + +**Cut, by default:** + +- Maintainer-note sidebars about labels / platform attribution unrelated to the bug surface. +- Bare-output reproductions when the load-bearing evidence is in a different command's output. +- File:line citations of source code already findable via the cited PR. +- Closing "if this verification is wrong, please reopen…" boilerplate. +- Redundant verbal framing of what the evidence already shows ("the table above proves…"). +- "Verification mode" pleasantries beyond one factual line. + +**Mandatory cap caveat.** When the score is capped (Step 9 baseline-validation gating, or any Step 11 degraded-mode path), the rendered Verdict section must include a one-line caveat naming the cap and the reason. Example: `Capped at 84 because Step 9's baseline-validation gate did not run (sandbox-build rot on v0.0.18: Dockerfile symlink layer removed by #2227).` Don't make readers reverse-engineer why the score didn't go higher — name it. + +**Mandatory hardware-substitution caveat.** When the issue carries `Platform: DGX Spark` or `Platform: GB10` and Step 7 provisioned a Brev SKU that is not the same silicon (Brev's stoppable GPU catalog is x86 + discrete H100/A100/L40S/T4 — not Grace Hopper / GB10 unified-memory ARM64), the rendered comment must include a one-line "Hardware substitution" note. Example: `Hardware substitution: verified on Brev n1-standard-4:nvidia-tesla-t4 (x86_64 + T4) as a substitute for the reporter's DGX Spark (ARM64 + GB10). For silicon-shape bugs (perf, memory architecture, drivers) this is not a faithful repro — please confirm on actual DGX Spark.` This goes in the metadata block right after `Verification mode:` so it's visible at the top, not buried in the analysis. + +**Mandatory `Verification mode` header line.** All three templates below include a `**Verification mode:**` line in the metadata block, naming what we did and didn't actually run (e.g., "runtime reproduction on Brev <SKU>; baseline + latest both installed and run" for the standard template; "static analysis at the verified-on tag — no runtime reproduction" for the by-design template; "runtime reproduction on Brev <SKU>; bug confirmed live on latest" for still-reproduces). Reader should never have to guess whether the verdict came from real install logs or from static analysis. + +**Link-pass self-verification (all templates).** Same rule as Step 8.5d's link pass, applied to every template. Resolve at least one rendered markdown link from each section that has them (`What's structurally fixed` / `Vestigial references` / `Existing CI coverage` for by-design; `Relevant changes since` / transcript code-anchor citations for the standard template) via `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 if path exists at tag, 404 otherwise) or `curl -fsI <blob-url>`. A 404 on a citation in the rendered comment is worse than no citation — it advertises verification work that didn't actually happen. If any link fails to resolve, fix it or bail to `verify-inconclusive`. + +**Mandatory closing block — reporter @-mention with confirmation language.** Every template below ends with an explicit @-mention of the original reporter using this exact shape: + +> @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. + +The skill cannot independently confirm a closed-as-fixed verdict — only the reporter knows whether their original symptom is gone in their environment. The @-mention is what converts a "skill says it's fixed" claim into actionable confirmation work for QA. Customize `<Z>` per case (the version that shipped the fix or `$LATEST`), but never omit the line. + +**Mandatory unanswered-question prefix and dual @-mention.** When Step 3 sets `UNANSWERED_MAINT_LOGIN` (a maintainer's question is older than 7 days and the reporter never replied), the verdict comment changes shape in two places: + +1. **Prepend a lead paragraph** as the very first line of the body, before the `## Stale-issue verification` heading. The lead paragraph is a single line: + + ```text + [@UNANSWERED_MAINT_LOGIN's comment](UNANSWERED_MAINT_URL) from UNANSWERED_MAINT_DATE is still unanswered. Posting independent verification below to unstick the thread. + ``` + + …with the bracketed variables expanded from the values exported by Step 3. + +2. **Replace the closing reporter-only @-mention with a dual @-mention** that names BOTH the maintainer (acknowledging the open question) and the reporter (per the standard confirmation pattern): + + > @\<UNANSWERED_MAINT_LOGIN\> — flagging that your question above is still open; the verification below may answer it. @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. + +This applies to all three templates (fixed, still-reproduces, by-design). The skill becomes the *unsticking voice* on a thread that has gone quiet — never a clueless interruption when discussion is fresh (Step 3 already filtered the within-7-day case). + +**Comment template (fixed / inconclusive — bug not reproduced on latest):** + +````markdown +## Stale-issue verification — automated + +**Reported on:** v0.0.31 +**Verified on:** v0.0.34 (commit abc1234) +**Verification mode:** runtime reproduction on Brev `<instance-class>` — baseline (v0.0.31) and latest (v0.0.34) both installed and run; comparison made on the captured transcripts. (Or: "runtime reproduction on Brev `<instance-class>` — baseline-install-skipped (`.openclaw-data` rot, see Step 11), latest-only run; verdict capped at 84.") +**Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 / <CUDA version if GPU> + +### Baseline (reported version) + +- Install: succeeded · skipped (install rotted) +- Reproducer: extracted verbatim · synthesized (−30 penalty) +- Result: bug symptom matched (validated) · could not validate (skipped Step 8c gate) + +<details><summary>Baseline transcript</summary> + +```text +<full baseline transcript> +``` + +</details> + +### Latest + +- Install: succeeded +- Result: not reproducible — clean run, no bug symptom observed + +<details><summary>Latest transcript</summary> + +```text +<full latest transcript> +``` + +</details> + +### Verdict + +**Confidence:** 88 / 100. Labelling `fixed-on-latest`. + +<details><summary>Relevant changes since v0.0.31</summary> + +- abc1234 — fix: <commit subject> +- def5678 — refactor: <commit subject> + +</details> + +@<reporter> — please confirm the symptom is gone on a recent build (≥ v0.0.<Z>) and reopen with a fresh reproducer if you observe otherwise. + +<!-- nemoclaw-verify-stale v1 2026-05-12 --> +```` + +**Comment template (still reproduces — Step 9 special case):** + +````markdown +## Stale-issue verification — still reproducible + +**Reported on:** v0.0.31 +**Verified on:** v0.0.34 (commit abc1234) +**Verification mode:** runtime reproduction on Brev `<instance-class>` — baseline confirmed the symptom matches the issue; latest (v0.0.34) also produced the symptom. Bug is still live. +**Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 + +The skill ran the reported reproducer on v0.0.34 and observed the same bug symptom described in this issue. The bug is still live. + +No label applied. Will re-verify automatically next weekly run; if a fix lands in the interim, the next pass catches it. + +@<reporter> — please confirm the symptom still matches your observation on v0.0.<Y> and reopen with any updated reproducer or environment details if it has shifted. + +<details><summary>Baseline transcript (validated reproducer)</summary> + +```text +<baseline transcript> +``` + +</details> + +<details><summary>Latest transcript (bug still observed)</summary> + +```text +<latest transcript> +``` + +</details> + +<!-- nemoclaw-verify-stale v1 2026-05-12 --> +```` + +The trailing HTML comment is the **idempotency marker** Step 3 looks for. Always include today's date in `YYYY-MM-DD` format so the candidate filter can apply the 7-day TTL. + +**Pre-post state-check.** A long-running verification can race with the maintainer closing the issue independently — happened on #2513 and #2519 (mid-batch closes by @jyaunches with their own verification). Re-check `state == OPEN` right before posting. If closed, apply the label tag-only (skipping the comment, since the maintainer's own close-comment is now the authoritative record) and skip the Project 199 move. + +```bash +STATE=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json state --jq .state) +if [ "$STATE" != "OPEN" ]; then + echo "[verify-stale] #$ISSUE_NUMBER closed since verification started — applying label tag-only, skipping comment + tracker move" + gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "$LABEL" + exit 0 +fi +``` + +**Post the comment and apply the label:** + +```bash +gh issue comment "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --body-file comment.md +gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "fixed-on-latest" +# or for <60: +# gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "verify-inconclusive" +``` + +**Move the issue to "Needs Review" on the NemoClaw Development Tracker (only on `fixed-on-latest`).** The tracker is GitHub Project [NVIDIA/199](https://github.com/orgs/NVIDIA/projects/199) ("NemoClaw Development Tracker"). When the skill's verdict is `fixed-on-latest`, the issue moves to **Needs Review** so the maintainer queue picks it up for confirmation; after the reporter confirms and the maintainer closes, existing Project automation (or a manual move) advances it to Done. **No move on `wontfix` / `verify-inconclusive` / no-label-still-reproduces** — those have separate close paths. + +This step requires the `project` scope on the maintainer's gh CLI (`gh auth refresh -h github.com -s project` in a real terminal once; OAuth device-code flow). If the scope is missing, the lookup query returns an auth error — fall through with a one-line warning rather than failing the whole run. + +```bash +# Project 199 constants (re-run gh project field-list 199 --owner NVIDIA --format json +# if the project gets renamed/restructured and these IDs drift): +PROJECT_ID="PVT_kwDOABpemM4BSCP5" +STATUS_FIELD_ID="PVTSSF_lADOABpemM4BSCP5zg_r9p8" +NEEDS_REVIEW_OPTION_ID="5c5922a9" + +# Only fire on fixed-on-latest. Skip silently otherwise. +if [ "$VERDICT" = "fixed-on-latest" ]; then + # Find the issue's existing project item, if any. + ITEM_ID=$(gh api graphql -f query=' + query($num: Int!) { + repository(owner: "NVIDIA", name: "NemoClaw") { + issue(number: $num) { + projectItems(first: 10) { + nodes { id project { number } } + } + } + } + }' -F num="$ISSUE_NUMBER" \ + --jq '.data.repository.issue.projectItems.nodes[] | select(.project.number == 199) | .id' \ + 2>/dev/null | head -1) + + # If the issue isn't on the project yet, add it. (NV QA bots usually add new + # issues automatically, but cover the gap.) + if [ -z "$ITEM_ID" ]; then + ITEM_ID=$(gh project item-add 199 --owner NVIDIA \ + --url "https://github.com/NVIDIA/NemoClaw/issues/$ISSUE_NUMBER" \ + --format json --jq .id 2>/dev/null) + fi + + if [ -n "$ITEM_ID" ]; then + gh project item-edit \ + --id "$ITEM_ID" \ + --project-id "$PROJECT_ID" \ + --field-id "$STATUS_FIELD_ID" \ + --single-select-option-id "$NEEDS_REVIEW_OPTION_ID" \ + >/dev/null && echo "[verify-stale] moved #$ISSUE_NUMBER to 'Needs Review' on Project 199" + else + echo "[verify-stale] WARN could not resolve project item for #$ISSUE_NUMBER on Project 199 — label applied but tracker not moved" + fi +fi +``` + +The Step 12 activity log line should record the project move (or the warn-and-skip case) so a maintainer scanning the log can spot tracker drift. Add a `Tracker:` row to the per-issue entry: `Tracker: moved to Needs Review` | `not moved (verdict: <X>)` | `not moved (project lookup failed)`. + +--- + +## Step 11: Infra Failure Handling + +Two different failure types, two different responses. + +**Latest-install failure** (Step 8d) or reuse-check / provisioning / harness errors: hard infra failure. + +- Print the error. +- Apply **no label** — infra failures must not pollute the verification record. +- Post a short comment **only if explicitly requested by the invoking user**. Default is silent move-on. +- Continue to the next candidate in batch mode. + +The next weekly run retries naturally. + +**Baseline-install failure** (Step 8a, reported version won't install on a modern image): not a hard failure — degraded mode. + +- Set `BASELINE_INSTALL_FAILED=1`, skip 8b/8c, jump to 8d. +- Step 9 applies the score cap (max 84) — corroboration signals raise the score within the cap but cannot lift past it. +- Note "baseline-install-skipped" in the final comment so a reviewer knows the verification ran without the script-validation gate. + +**Baseline-build failure** (Step 8a binary install succeeded, but the in-image `Dockerfile` build during sandbox creation failed on a layer that was structurally removed in a later release): also degraded mode, distinct from binary install rot. Surfaced during the #2007 e2e run on v0.0.18 (`/sandbox/.openclaw-data/workspace/media` symlink layer, removed entirely by #2227). + +- Set `BASELINE_INSTALL_FAILED=1` (same flag — Step 9's cap-at-84 rule keys off it regardless of which phase rotted). +- Skip 8b/8c, jump to 8d. +- Note "baseline-build-skipped" in the final comment with the specific failing layer/file so a reviewer can see *why* the v0.0.X image no longer builds (the why is usually a follow-on PR that removed the rotted layer). +- Do not retry the build with a patched Dockerfile — that breaks faithfulness. We're claiming "couldn't independently re-trigger the original symptom on baseline," not "we made the old version work somehow." + +Both baseline-rot variants share the same downstream effect: Step 9 cap, Step 10 caveat, @-mention reporter to confirm. Distinguishing them in the comment helps a reviewer understand the failure mode without re-running. + +This degradation is expected — old releases rot at multiple phases (binary installer URL drift, base-image dependencies vanish, in-image Dockerfile layers get removed by structural refactors). We still want to extract whatever signal we can from the latest run plus PR/commit evidence, just at a more conservative confidence ceiling. + +**Empirical reality after two e2e runs:** baseline-build-rot is the **dominant** failure mode for any reported version more than ~5–7 patches behind, not an edge case. Both #2007 (v0.0.18, 17 patches behind) and #2592 (v0.0.28, 7 patches behind) hit it. The cap-at-84 with reporter @-mention is the **modal** verdict shape for stale-issue verification, not the exception. Reframe expectations accordingly: + +- For issues reported >5 patches behind `$LATEST`, plan for the cap-at-84 path. Pre-flight (PR-search, pickaxe) carries more weight than baseline runtime evidence. +- For issues reported within 1–4 patches of `$LATEST`, baseline is more likely to install cleanly and the full +50 path is reachable. +- The skill's design assumes baseline + latest both run cleanly; in practice latest-only with cap-at-84 is the workhorse path. The score-cap is doing real work, not just a fallback. + +**Keep-box-on-inconclusive.** When `verify-inconclusive` lands (Step 8c gave up, or Step 9 score < 60), **skip the cleanup trap** for this run if the box was provisioned by this run — set `PROVISIONED_NEW=0` before the trap fires so the EXIT handler is a no-op. Print the `brev shell "$INSTANCE_NAME"` command and an explicit `brev delete "$INSTANCE_NAME"` reminder in the run output so the maintainer can triage and clean up manually. Reused boxes stay regardless. Ship-failed verifications are the exact case where having an inspectable artifact pays for itself; an unbounded sleep-and-delete in the background isn't reliable across session ends, so we leave deletion explicit. + +--- + +## Step 12: Log to Activity + +After each issue (verified, inconclusive, by-design, or infra-failed), append to `${VERIFY_STALE_LOG_DIR:-$HOME/development/daily-rhythm/activity}/nemoclaw-verify-stale-log.md`. The default path matches the personal-organizer convention; export `VERIFY_STALE_LOG_DIR` to point elsewhere (CI, shared volume, etc.). Create the directory if missing — do not assume it exists. + +```markdown +### NVIDIA/NemoClaw#<number> — <title> +**Date:** YYYY-MM-DD +**Reported on:** v0.0.31 +**Verified on:** v0.0.34 +**Environment:** CPU | GPU (<instance type>) +**Box:** reused <name> | provisioned <name> | local (no Brev — Step 6.7 short-circuit) +**Baseline install:** succeeded | failed (degraded mode) +**Baseline match:** validated (verbatim) | validated (synth) | failed (verify-inconclusive) | skipped +**Latest install:** succeeded | failed (infra error) +**Latest result:** not-reproduced (clean) | still-reproduces | partial / flake | n/a (skipped 8d) +**Confidence:** 88 / 100 | n/a (still-reproduces) +**Label applied:** fixed-on-latest | verify-inconclusive | status: wont-fix | none (still-reproduces) | none (infra) +**Tracker:** moved to Needs Review on Project 199 | not moved (verdict: <X>) | not moved (project lookup failed) +**Brev wall time (approx):** N min + +--- +``` + +Create the file if missing, with this header: + +```markdown +# NemoClaw — Verify Stale Log + +A running record of stale-issue verification runs on NVIDIA/NemoClaw. +Persisted via daily-rhythm to GitLab. + +--- +``` + +At end of a batch session, prepend a session summary: + +```markdown +## YYYY-MM-DD — Verify Session +**Issues considered:** N +**Verified `fixed-on-latest`:** N +**Marked `status: wont-fix` (by-design path):** N +**Marked `verify-inconclusive`:** N +**Local-first short-circuits (no Brev cost):** N +**Skipped (Windows / macOS / integration / no version):** N +**Infra failures:** N +**Brev wall time:** N min · approx $X.XX + +--- +``` + +Never stage or commit the log to the NemoClaw repo. + +--- + +## Cadence + +- **Weekly cron** — Monday morning, batch mode, ≤15 issues (the Step 1 cap, sliced after Step 3/4 filters). +- **Manual** — invoke with a single issue number anytime. + +--- + +## Out of Scope (v1) + +- Auto-closing issues. Always tag-only; a human pulls the trigger. +- macOS verification *via the Brev path*. Brev offers no macOS instances. The Step 6.7 local-first short-circuit *does* run on a maintainer's macOS laptop — so manual single-issue runs against pure-CLI bugs work on macOS. The weekly batch cron is Linux-only because that path always uses Brev. +- Issues requiring third-party integration credentials (Slack, Discord, Telegram, Hermes, OpenClaw, WeChat). +- Service-account bot identity. v1 runs under each maintainer's own GitHub credentials. +- Versioned labels. A single `fixed-on-latest` label is swept on each release cut. + +--- + +## Companion Behavior + +`nemoclaw-maintainer-cut-release-tag` sweeps `fixed-on-latest` and `verify-inconclusive` from all open issues at release time. Without that sweep, "latest" drifts and verifications go stale silently. The by-design path uses the existing repo `status: wont-fix` label; that label is **not** swept (it's also applied for non-skill reasons such as scope or priority decisions, and clearing it would erase human triage work). From a78126baa7f1665ea7864c07829805b6bb565efe Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Sun, 10 May 2026 20:35:54 -0700 Subject: [PATCH 35/40] fix(verify-stale): address three CodeRabbit findings on PR #3327 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - cut-release-tag: bump projectItems(first: 10) to first: 100 in the Project 199 Done-state check, so an issue attached to >10 projects doesn't silently bypass the verification-label sweep when 199 falls past the first page. - execution-and-comment: scope the mandatory reporter @-mention rule to "Every template below except Still-reproduces". Resolves the contradiction with the per-verdict table (still-reproduces is intentionally minimal — no @-mention, no transcripts). - verify-stale: drop the standalone "prefer --skip-browser/--token in non-TTY" sentence after the brev auth recipe. The recipe block already lists the browser flow as primary with headless as fallback; the standalone line contradicted that ordering. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md | 2 +- .agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md | 2 -- .../reference/execution-and-comment.md | 2 +- 3 files changed, 2 insertions(+), 4 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md index eba035fea7c..36ea0c754f0 100644 --- a/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md @@ -160,7 +160,7 @@ for label in fixed-on-latest verify-inconclusive; do query($num: Int!) { repository(owner: "NVIDIA", name: "NemoClaw") { issue(number: $num) { - projectItems(first: 10) { + projectItems(first: 100) { nodes { project { number } fieldValueByName(name: "Status") { diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index 0f043abcb9d..e06b36230bd 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -450,8 +450,6 @@ curl -fsI "$INSTALL_URL" >/dev/null 2>&1 || { } ``` -If invoked from an environment without a TTY (some agent harnesses), prefer `brev login --skip-browser` or `--token` over the default browser flow. - --- ## Step 6.7: Try Local Reproduction First diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md index 3c775c496e0..82a670d4a6a 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md @@ -867,7 +867,7 @@ Patterns live in a fenced block (not a markdown table) because patterns 8 and 9 **Link-pass self-verification (all templates).** Same rule as Step 8.5d's link pass, applied to every template. Resolve at least one rendered markdown link from each section that has them (`What's structurally fixed` / `Vestigial references` / `Existing CI coverage` for by-design; `Relevant changes since` / transcript code-anchor citations for the standard template) via `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 if path exists at tag, 404 otherwise) or `curl -fsI <blob-url>`. A 404 on a citation in the rendered comment is worse than no citation — it advertises verification work that didn't actually happen. If any link fails to resolve, fix it or bail to `verify-inconclusive`. -**Mandatory closing block — reporter @-mention with confirmation language.** Every template below ends with an explicit @-mention of the original reporter using this exact shape: +**Mandatory closing block — reporter @-mention with confirmation language.** Every template below **except `Still-reproduces`** ends with an explicit @-mention of the original reporter using this exact shape: > @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. From 07bdcdb51cfa487c7fec5dbfc6e4d0a23467662a Mon Sep 17 00:00:00 2001 From: Test User <test@example.com> Date: Mon, 11 May 2026 11:18:10 -0700 Subject: [PATCH 36/40] feat(verify-stale): self-assign issue to maintainer on fixed-on-latest verdict MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When the skill moves an issue to "Needs Review" on Project 199, also assign the issue to the maintainer who ran the skill ($GH_IDENTITY from Step 6.5). The assignment puts the verification record in the maintainer's personal review queue — they staked their name on the verdict, the issue should sit with them until they close it or the reporter responds. Gated to fixed-on-latest only — same gate as the tracker move. status: wont-fix / verify-inconclusive / no-label runs don't self-assign (their close paths are different). Step 12 activity log gets a new Assignee: row to record the action. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --- .../reference/execution-and-comment.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md index 82a670d4a6a..5291c668a63 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md @@ -999,7 +999,7 @@ gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "fixed-on-lates # gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "verify-inconclusive" ``` -**Move the issue to "Needs Review" on the NemoClaw Development Tracker (only on `fixed-on-latest`).** The tracker is GitHub Project [NVIDIA/199](https://github.com/orgs/NVIDIA/projects/199) ("NemoClaw Development Tracker"). When the skill's verdict is `fixed-on-latest`, the issue moves to **Needs Review** so the maintainer queue picks it up for confirmation; after the reporter confirms and the maintainer closes, existing Project automation (or a manual move) advances it to Done. **No move on `wontfix` / `verify-inconclusive` / no-label-still-reproduces** — those have separate close paths. +**Move the issue to "Needs Review" on the NemoClaw Development Tracker AND self-assign (only on `fixed-on-latest`).** The tracker is GitHub Project [NVIDIA/199](https://github.com/orgs/NVIDIA/projects/199) ("NemoClaw Development Tracker"). When the skill's verdict is `fixed-on-latest`, the issue moves to **Needs Review** AND the issue is assigned to the maintainer who ran the skill (`$GH_IDENTITY` from Step 6.5) — assignment puts the issue in their personal review queue so they don't lose track of what they've staked their name on. After the reporter confirms and the maintainer closes, existing Project automation (or a manual move) advances it to Done. **No move and no assign on `wontfix` / `verify-inconclusive` / no-label-still-reproduces** — those have separate close paths. This step requires the `project` scope on the maintainer's gh CLI (`gh auth refresh -h github.com -s project` in a real terminal once; OAuth device-code flow). If the scope is missing, the lookup query returns an auth error — fall through with a one-line warning rather than failing the whole run. @@ -1044,6 +1044,11 @@ if [ "$VERDICT" = "fixed-on-latest" ]; then else echo "[verify-stale] WARN could not resolve project item for #$ISSUE_NUMBER on Project 199 — label applied but tracker not moved" fi + + # Self-assign the issue to the maintainer who ran the skill — puts it in their + # personal review queue alongside the Needs Review state. + gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-assignee "$GH_IDENTITY" \ + >/dev/null && echo "[verify-stale] assigned #$ISSUE_NUMBER to @$GH_IDENTITY" fi ``` @@ -1109,6 +1114,7 @@ After each issue (verified, inconclusive, by-design, or infra-failed), append to **Confidence:** 88 / 100 | n/a (still-reproduces) **Label applied:** fixed-on-latest | verify-inconclusive | status: wont-fix | none (still-reproduces) | none (infra) **Tracker:** moved to Needs Review on Project 199 | not moved (verdict: <X>) | not moved (project lookup failed) +**Assignee:** @<GH_IDENTITY> | not assigned (verdict: <X>) **Brev wall time (approx):** N min --- From 4962da52e7d4dc7ff120ead0cf2b239e9efec8e5 Mon Sep 17 00:00:00 2001 From: Carlos Villela <cvillela@nvidia.com> Date: Sat, 23 May 2026 11:22:04 -0700 Subject: [PATCH 37/40] refactor(skills): split verify-stale references Signed-off-by: Carlos Villela <cvillela@nvidia.com> --- .../nemoclaw-maintainer-verify-stale/SKILL.md | 530 +------- .../reference/brev-provisioning.md | 297 +++++ .../reference/by-design.md | 178 +++ .../reference/candidate-selection.md | 238 ++++ .../reference/environment-and-reproducer.md | 260 ++++ .../reference/execution-and-comment.md | 1173 ----------------- .../reference/reproduction-rubrics.md | 248 ++++ .../reference/scoring-comments-and-logging.md | 496 +++++++ 8 files changed, 1764 insertions(+), 1656 deletions(-) create mode 100644 .agents/skills/nemoclaw-maintainer-verify-stale/reference/brev-provisioning.md create mode 100644 .agents/skills/nemoclaw-maintainer-verify-stale/reference/by-design.md create mode 100644 .agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md create mode 100644 .agents/skills/nemoclaw-maintainer-verify-stale/reference/environment-and-reproducer.md delete mode 100644 .agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md create mode 100644 .agents/skills/nemoclaw-maintainer-verify-stale/reference/reproduction-rubrics.md create mode 100644 .agents/skills/nemoclaw-maintainer-verify-stale/reference/scoring-comments-and-logging.md diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md index e06b36230bd..1712aa2203e 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/SKILL.md @@ -1,6 +1,6 @@ --- name: nemoclaw-maintainer-verify-stale -description: Verify whether old NVIDIA/NemoClaw bug reports still reproduce against the latest tag. Picks candidate issues opened against older versions, runs the reproducer locally first when possible (Linux or macOS), otherwise reuses or provisions a Brev Linux box (CPU or GPU), detects behavior that was intentionally changed, scores confidence, and posts an evidence-backed comment with a label (fixed-on-latest, status: wont-fix, or verify-inconclusive). Tag-only — never auto-closes. Brev verification is Linux-only in v1; Windows and integration-token-dependent issues are skipped. Trigger keywords - verify stale, verify fixed, reproduce on latest, stale issue, old bug, fixed-on-latest, status: wont-fix, verify-inconclusive, drain backlog, brev verify. +description: "Verifies whether stale NVIDIA/NemoClaw bug reports still reproduce on the latest tag. Use when maintainers ask to verify stale issues, reproduce old bugs on latest, drain the bug backlog, or apply fixed-on-latest, verify-inconclusive, or status: wont-fix. Runs candidate filtering, local/Brev reproduction, by-design detection, confidence scoring, redacted comments, and tag-only labeling; never auto-closes." user_invocable: true --- @@ -9,489 +9,53 @@ user_invocable: true # NemoClaw Maintainer — Verify Stale Issues -Automates the manual loop of "spin up a Brev box, install latest NemoClaw, try to reproduce an old bug, comment with findings." Drains the bug backlog by surfacing issues that have been silently fixed. +Automates the maintainer loop: choose an old bug, verify whether it still reproduces on the latest NemoClaw tag, then post an evidence-backed comment and label. It is tag-only: never close issues automatically. -This skill is the outbound counterpart to `nemoclaw-diagnosis` (which files issues from CI failures). Diagnosis fills the queue; this drains it. +## Progress checklist ---- - -## Step 1: Determine Mode - -**Single-issue mode** — user provides an issue number: - -```bash -gh issue view <number> --repo NVIDIA/NemoClaw \ - --json number,title,body,labels,url,author,createdAt,comments -``` - -**Batch mode** — user says "batch", "weekly", or provides no number. Cap at **15 issues** for *processing* per run, enforced as a slice after Step 3/4 filters narrow the pool. The cap exists because batch is sequential (Step 7 reuse-or-provision keeps it on 1–2 Brev boxes total) and the wallclock budget is ~2–3 hours per 15-issue run; running larger forces the maintainer to either drop the per-plan approval gate or spread the batch across multiple sessions. - -The discovery query needs to see the entire open-bug pool — the per-run processing cap is downstream. Use `--limit 1000` so the skill doesn't silently drop issues beyond the page (the candidate triage run found 129 open bugs; an earlier `--limit 100` would have missed 29 of them). - -```bash -gh issue list --repo NVIDIA/NemoClaw --state open --limit 1000 \ - --label bug \ - --json number,title,body,labels,url,author,createdAt,comments -``` - -In batch mode, work through items one at a time. Present each verification plan and wait for approval before any Brev provisioning. - ---- - -## Step 2: Detect the Latest NemoClaw Version - -Try GitHub releases first; fall back to the highest semver tag from the GitHub API if no release is published. NemoClaw currently tags but does not publish releases, so the fallback is the load-bearing path today. Use `gh api` rather than `git ls-remote` so the skill works regardless of SSH key setup, and reuses the auth `gh` already has. - -```bash -LATEST=$(gh release view --repo NVIDIA/NemoClaw --json tagName -q .tagName 2>/dev/null) - -if [ -z "$LATEST" ]; then - LATEST=$(gh api repos/NVIDIA/NemoClaw/tags --paginate --jq '.[].name' \ - | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' \ - | sort -V | tail -1) -fi - -echo "Latest tag: $LATEST" -``` - -This is the version the skill will verify against. Record it — every comment must cite it. - ---- - -## Step 3: Filter Candidates - -Apply these rules in order. Drop any issue that fails a rule. - -**Issue-type allowlist:** must have `bug` label. -**Issue-type skip:** drop if any label exactly matches `documentation`, `status: wont-fix`, `status: needs-info`, `security`, OR is `enhancement` / starts with the prefix `enhancement:` (the repo has 8 prefixed variants — `enhancement: feature`, `enhancement: MCP`, `enhancement: testing`, `enhancement: ui`, `enhancement: provider`, `enhancement: platform`, `enhancement: policy`, `enhancement: inference`, `enhancement: integration`, `enhancement: performance`, `enhancement: skill` — and exact-match misses them all; surfaced from #1752). Use the canonical repo label names — bare `wontfix` / `needs-info` are NOT the repo's labels (verified via `gh label list`); the actual labels carry a `status:` prefix and a hyphen. - -**Platform skip (Brev-reproducible only in v1):** drop if any of `Platform: Windows/WSL`, `Platform: MacOS`, `Platform: macOS`, `Platform: Jetson AGX Thor/Orin`. Brev has no equivalent hardware for Jetson (embedded/edge ARM with integrated GPU is not in the Brev SKU catalog), so any Brev verification of a Jetson-only bug would produce a misleading "fixed-on-x86" verdict. Keep `Platform: Ubuntu`, `Platform: DGX Spark`, `Platform: GB10`, `Platform: All`, or no platform label. `Platform: DGX Spark` and `Platform: GB10` stay in scope but Step 10 requires a "Hardware substitution" caveat in the comment naming the Brev SKU we used as a substitute (Brev x86 GPU SKUs are not faithful to GB10 / Grace Hopper silicon for performance-shape or memory-architecture-shape bugs). - -**TUI / interactive-UI skip:** drop if the issue title contains `TUI`, `dashboard UI`, `chat UI`, `keystroke`, or `key press`, OR if the body describes interactive UI behavior (key sequences, mouse interactions, browser-side UI state) without a non-interactive reproducer (no `NEMOCLAW_NON_INTERACTIVE=1` or equivalent env var pattern). `brev exec` does not allocate a real TTY by default, so TUI reproducers hang or silently fail at the first prompt; v1 documents this as out-of-scope rather than emitting a wrong verdict. v1.1 may add a `script(1)` / `expect` / `tmux send-keys` harness to lift this skip. - -**Integration skip (deferred to v2):** drop if any of `Integration: Slack`, `Integration: Discord`, `Integration: Telegram`, `Integration: Hermes`, `Integration: OpenClaw`, `Integration: WeChat`. These need third-party credentials a fresh Brev box cannot provide. - -**Component allowlist (must have at least one):** `NemoClaw CLI`, `Sandbox`, `OpenShell`, `Docker`, `Getting Started`, or any `Platform:` label that survived the platform skip. - -**Idempotency:** drop if **either** of these is true: - -- The issue carries a `fixed-on-latest` or `verify-inconclusive` label. (Cleared by the release sweep in `nemoclaw-maintainer-cut-release-tag` so the issue re-opens on each release.) The by-design path uses the existing repo `status: wont-fix` label, which is already covered by the issue-type skip rule above — no separate idempotency clause needed for that path. -- A comment matching `<!-- nemoclaw-verify-stale v\d+ YYYY-MM-DD -->` was posted **within the last 7 days**. The regex matches any marker version (`v1`, `v2`, …) so future skill versions can re-verify older-marked issues by tightening the regex (e.g. require a specific marker version). The marker carries a date so the candidate filter can apply a TTL — useful for the still-reproduces case (Step 9), where no label is applied and we want next week's run to re-verify rather than skip forever. - -Implementation — match the marker against each comment's `createdAt`. Use `gh issue view --json comments` (single-issue mode already fetches this; batch mode's `gh issue list` also returns the comment array per issue): - -```bash -# Cutoff for the 7-day TTL. macOS and Linux date(1) syntax differ; try both. -SEVEN_DAYS_AGO=$(date -u -v-7d +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \ - || date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ) - -# Returns the timestamp of the most recent marker comment within the TTL, or empty. -RECENT_MARKER=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ - --jq --arg cutoff "$SEVEN_DAYS_AGO" ' - .comments[] - | select(.body | test("<!-- nemoclaw-verify-stale v\\d+ \\d{4}-\\d{2}-\\d{2} -->")) - | select(.createdAt > $cutoff) - | .createdAt' \ - | head -1) - -if [ -n "$RECENT_MARKER" ]; then - echo "Skip: marker posted $RECENT_MARKER (within 7-day TTL)" - # In single-issue mode: exit 0 with a friendly message. - # In batch mode: continue to the next candidate. -fi -``` - -Run this check for every candidate that survived the label-based filters above; drop those whose `RECENT_MARKER` is non-empty. - -**Unanswered-maintainer-question handling.** Find the most recent maintainer (`MEMBER`, `OWNER`, `COLLABORATOR`) comment that **looks like a question** (`?`, polite imperative like "please confirm/share/clarify", or starter like "could you / can you / do you") AND that the reporter has not replied to since. Pure triage acknowledgments (`"✨ Thanks for reporting…"`) are skipped. The age of the qualifying comment determines skip-or-proceed: - -- **Within 7 days:** **skip the issue** — the discussion is active, the skill running on top would conflict with the maintainer's framing or confuse the reporter. Surfaced during pre-flight on #2757; running verify-stale on top of a fresh "let me clarify what you observed" question from @cjagwani would have stomped on that conversation. -- **Older than 7 days:** **proceed with verification, but use the unanswered-question comment variant.** After 7 days the maintainer's question has either been forgotten or the reporter has dropped the ball; an independent skill verdict becomes the *unsticking voice* rather than a clueless interruption. The comment leads with a markdown link to the maintainer's unanswered comment (shape shown in the Step 10 template below) and @-mentions BOTH the maintainer and the reporter, not just the reporter. Reuse `$SEVEN_DAYS_AGO` from the marker-TTL check above. - -```bash -REPORTER=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json author --jq .author.login) - -# Most recent unanswered maintainer comment that looks like a question — filters out triage acknowledgments (#1642 surfaced this). -UNANSWERED_MAINT=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ - --jq --arg reporter "$REPORTER" --arg cutoff "$SEVEN_DAYS_AGO" ' - (.comments - | map(select((.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR") - and (.body | test("\\?|(?i)\\bplease (confirm|share|provide|clarify|tell|verify|check|let me know|let us know)|(?i)\\b(could|can|would) you\\b|(?i)\\bdo you (have|know|see|use)\\b")))) - | sort_by(.createdAt) | last) as $maint - | if $maint == null then null - else - ((.comments - | map(select(.author.login == $reporter and .createdAt > $maint.createdAt)) - | length) as $replies - | if $replies > 0 then null - else { - createdAt: $maint.createdAt, - url: $maint.url, - login: $maint.author.login, - recent: ($maint.createdAt > $cutoff) - } - end) - end') - -if [ -n "$UNANSWERED_MAINT" ] && [ "$UNANSWERED_MAINT" != "null" ]; then - MAINT_RECENT=$(printf '%s' "$UNANSWERED_MAINT" | jq -r .recent) - MAINT_DATE=$(printf '%s' "$UNANSWERED_MAINT" | jq -r .createdAt) - MAINT_LOGIN=$(printf '%s' "$UNANSWERED_MAINT" | jq -r .login) - MAINT_URL=$(printf '%s' "$UNANSWERED_MAINT" | jq -r .url) - - if [ "$MAINT_RECENT" = "true" ]; then - echo "Skip: active maintainer discussion (unanswered comment from @$MAINT_LOGIN at $MAINT_DATE, within 7 days)" - # Single-issue mode: exit 0 with the message; batch mode: continue to next candidate. - else - echo "[verify-stale] proceeding with unanswered-question variant — @$MAINT_LOGIN's comment from $MAINT_DATE is older than 7 days" - # Step 10's comment template will lead with the unanswered-question prefix and @-mention - # both the maintainer and the reporter. Export these for the templater: - export UNANSWERED_MAINT_LOGIN="$MAINT_LOGIN" - export UNANSWERED_MAINT_URL="$MAINT_URL" - export UNANSWERED_MAINT_DATE="$MAINT_DATE" - fi -fi -``` - -When the unanswered-question variant fires (`UNANSWERED_MAINT_LOGIN` set), Step 10's comment template prepends a lead paragraph (exact shape lives with the templates in Step 10), and the closing @-mention block names BOTH the maintainer (acknowledging their question) and the reporter (asking for confirmation per the standard pattern), instead of just the reporter. - -**Candidate rule:** keep the issue if **either**: - -- The reported version (parsed from body or labels — see Step 4) is **at least 2 versions behind** `$LATEST` in the rightmost-incrementing component, **or** -- The issue is **older than 7 days** AND a specific version is parseable from its body or labels. - -For NemoClaw's current `0.0.x` line, "rightmost-incrementing component" is the patch number — a v0.0.31 report against a v0.0.34 latest is 3 versions behind. Once NemoClaw moves to `0.1.x` or higher, the rule applies to the next-rightmost component instead. Pick whichever component is actively iterating. - ---- - -## Step 4: Parse Reported Version - -The regex is intentionally **release-line agnostic**. Today NemoClaw ships `v0.0.x`, but the same parser must keep working when it moves to `v0.1.x`, `v1.x.x`, or anything else. Don't hardcode the major/minor digits. - -Sources, in order of trust: - -1. **Labels.** Any label that exactly matches `^v\d+\.\d+\.\d+$` AND appears in the repo's tag list. Labels matching the regex but absent from tags (e.g. `v0.0.35` as a *release-target* milestone before that version ships) are roadmap markers, not "reported on" — drop them. -2. **Body.** Use a **proximity-anchored** regex: `(?i)nemoclaw[^a-z\n]{0,80}v?(\d+\.\d+\.\d+)`. This matches a version that follows `nemoclaw` within 80 non-letter, non-newline characters, capturing just the semver. The anchoring is load-bearing — without it the parser also picks up `openshell 0.0.4`, Node.js `v22.16.0`, IP addresses (`0.0.0.0:11434`, `127.0.0.1`), and other near-NemoClaw products that happen to share the `v0.0.x` line. (This was confirmed in the dry-run: a non-anchored parser produced 12 false-positive candidates whose smallest tag-valid version was actually OpenShell's, not NemoClaw's.) -3. **Comments by the original reporter** — same anchored regex as the body. - -Collect every match from sources 2 and 3 (a single body may mention multiple versions — `0.0.6 and v0.0.10`). Then validate. - -**Validate against the tag list.** A parsed version must exist as a real git tag, otherwise drop it. This single check kills four classes of error in one pass: - -- Reporter typos that cite a non-existent version (`v0.1.0` when only `v0.0.x` is released — observed 3× in the live backlog). -- Calver mistakes (`2026.3.11` — observed 1×). -- Future roadmap labels that slipped past source 1. -- Versions parsed from prose that happen to look semver-ish but aren't releases. - -```bash -gh api repos/NVIDIA/NemoClaw/tags --paginate --jq '.[].name' > /tmp/nemoclaw-tags.txt - -# For each candidate version V: -grep -Fxq "$V" /tmp/nemoclaw-tags.txt || drop_version "$V" -``` - -After validation, **pick the smallest surviving version** as the reported version (most conservative — it maximizes versions-behind). This handles "this bug was first reported on v0.0.6 and still happens on v0.0.10" cleanly: we verify against latest, and if the bug is gone, both reports are addressed. - -If no version survives, drop the issue from the candidate set — we cannot establish "previous version". - -**Variable format for downstream steps.** Set `REPORTED_VERSION` to the **full tag string** (e.g., `REPORTED_VERSION="v0.0.32"`), not just the patch number. Step 8a's installer expects the full tag via the `NEMOCLAW_INSTALL_TAG` env var. - -**Batch cap enforcement.** In batch mode, after Step 3 label filters and the Step 4 version+candidate-rule filters narrow the pool, sort surviving candidates by `(-versions_behind, -age_days)` so the most stale come first, then **slice to the top 15**: - -```bash -# Each candidate has at minimum: number, reported, behind, age_days -SLICED=$(printf '%s' "$CANDIDATES_JSON" | jq ' - sort_by([-(.behind // 0), -(.age_days // 0)]) - | .[0:15]') -SLICED_COUNT=$(printf '%s' "$SLICED" | jq 'length') -TOTAL=$(printf '%s' "$CANDIDATES_JSON" | jq 'length') -echo "Batch run: processing $SLICED_COUNT of $TOTAL eligible candidates (cap: 15)." -[ "$TOTAL" -gt 15 ] && echo " Spillover: $((TOTAL - 15)) candidates deferred to next run; the marker-comment TTL (Step 3) keeps them eligible." -``` - -The slice is the only enforcement of the cap — without it, "Cap at 15" is policy that nothing actually applies. Single-issue mode bypasses the cap entirely (the user explicitly named one issue). - -**NVBugs cross-reference.** Many NV QA bugs include an NVBugs ticket footer like `[NVB#6100043]`. Extract it at the same time as the version so Step 8.5's comment template (and any other comment template that wants to mention it) can include the cross-reference: - -```bash -NVBUGS_REF=$(printf '%s' "$BODY" | grep -oE '\[NVB#[0-9]+\]' | head -1) -``` - -Templates ignore this when empty. When present, the comment must note that closing the GitHub issue does not propagate to NVBugs and QA needs to update the ticket separately. - -### Implementer note: regex-pipeline pitfalls - -Three real failure modes surfaced during the v1 dry-run. Test each before trusting your implementation: - -1. **Empty-match handling.** A naive pipeline like `[scan(regex)] | first | .[0] | tonumber // fallback` silently dropped 9 real candidates (e.g. #2861 with `NemoClaw 0.0.32`, #2604 with `NemoClaw: 0.0.28`). When `scan` returns no matches, `[]` flows in, `first` returns null, `null | .[0]` errors, and `//` does not propagate cleanly through the error. Bind each pass to a named variable, coalesce at the end: - - ```text - primary := first nemoclaw-anchored match in body (or null) - result := primary ?? null - ``` - - Then explicitly test against a body with **no** version mention. - -2. **Capture-group consistency.** A regex without a capture group (e.g. `\bv\d+\.\d+\.\d+\b`) makes `scan` emit raw strings; with a capture group (e.g. `\b(v\d+\.\d+\.\d+)\b`), `scan` emits arrays. Mixing the two within one pipeline (`first | .[0]?`) works for one and silently fails for the other. Use capture groups consistently across all branches. - -3. **Variable scoping in `select(...)`.** A line like `select($tags | index(.))` rebinds `.` to `$tags` inside the parens, so `.` no longer refers to the surrounding label being checked. Bind first: `. as $lbl | select($tags | any(. == $lbl))`. Symptom in this dry-run: the future-release label `v0.0.35` passed validation that should have rejected it. - ---- - -## Step 5: Classify the Verification Environment - -**CPU vs GPU:** GPU if any of these signals are present, else CPU. - -- Labels: `Platform: GB10`, `Platform: DGX Spark`. -- Body keywords (whole-word, case-insensitive): `nvidia-smi`, `cuda`, `H100`, `A100`, `L40S`, `L4`, `T4`, `GB10`, `DGX`, `vllm`, `tensorrt`. Match as whole words — `inference` and `model serving` are too noisy (e.g. `models.providers.inference.baseUrl` is a config path on CPU bugs, not a GPU need) and intentionally excluded. - -CPU default keeps cost low. Only escalate to GPU when the reproducer needs one. - -**Bug class classification.** In addition to CPU/GPU, classify the bug's verification shape so Step 8 routes to the right rubric. Classes are mutually exclusive — pick the first that matches: - -| Class | Detection heuristic | Routes to | -|---|---|---| -| `performance` | Body or title mentions latency thresholds (`P50`, `P90`, `ms`, `seconds`, `slow`, `hangs`, `timeout` with a numeric value), or mentions `memory leak` / `over time` / `eventually` | Step 8e (multi-run distribution rubric) | -| `rebuild-cycle` | Body mentions `rebuild`, `recreate`, `restart`, `pod recreate`, `across rebuilds`, `after restart`, `survives a destroy` | Step 8f (run-rebuild-rerun harness) | -| `log-only` | Body's symptom is logs-not-stdout: `see lots of error in <X> log`, `os.networkInterfaces guard errors`, anything pointing at a specific log file rather than the reproducer's stdout/stderr | Step 8b's match rubric extended with log-scraping | -| `functional` (default) | Everything else — exit code + stdout/stderr matching | Step 8b standard rubric | - -Most bugs are `functional`. The other three classes need verification harnesses that the standard rubric can't produce honestly — e.g., one clean run of a perf reproducer doesn't tell you the p50 budget was met; one onboard run doesn't tell you a config survives a rebuild. Set `BUG_CLASS=<class>` so downstream steps can branch. - -**Provider classification.** Some bugs are tied to a specific inference provider (NVIDIA NIM, Gemini, Anthropic, OpenAI) and won't reproduce faithfully under Ollama substitution. Classify which provider the issue references so downstream steps either prompt for the right API key or accept the substitution penalty: - -| Detection signal | Provider | -|---|---| -| `Provider: NVIDIA` label, body mentions `NVIDIA NIM`, `build.nvidia.com`, `nvapi-...`, `NVIDIA_API_KEY`, or `NEMOCLAW_PROVIDER=build` | `nim` | -| `Provider: Gemini` label, body mentions `Gemini`, `gemini-flash`, `gemini-pro`, `GEMINI_API_KEY` | `gemini` | -| `Provider: Anthropic` / `Provider: AWS` (Bedrock) labels or matching keywords | `anthropic`/`bedrock` | -| `Provider: Ollama`, body mentions `ollama` or `NEMOCLAW_PROVIDER=ollama`, or no provider mentioned at all | `ollama` (default) | - -Set `BUG_PROVIDER=<provider>`. - -**Required-API-key prompt.** When `BUG_PROVIDER` is anything other than `ollama` AND the bug's reproducer actually exercises inference (not pure CLI surface or sandbox build), the skill MUST stop here and prompt the maintainer interactively before any Brev cost is incurred: +Copy this checklist and update it as you work: ```text -The reporter's reproducer uses the <provider> provider, which requires a real API key -to verify faithfully. Three options: - - 1. Provide an API key via file (NEVER on the command line — keys in argv are - visible in `ps -ef` to anyone with shell access on either machine). Write - the key to a 600-perm file on your laptop: - - printf '%s' '<your-key>' > ~/.nvidia-api-key - chmod 600 ~/.nvidia-api-key - - The skill copies the file to the Brev box via `brev copy` (encrypted SSH), - reads it inside the box with `NVIDIA_API_KEY=$(cat ~/.nvidia-api-key)`, - and never puts the value on a command line. Box deletion removes the file - from the box; you should `rm ~/.nvidia-api-key` on your laptop after the - run. - - 2. Substitute Ollama and accept the -30 confidence penalty (per Step 8a.5). The - verdict will be capped because we're not exercising the real provider's code - path. - - 3. Skip this issue. Mark `verify-inconclusive` with the reason "requires <provider> - API key — not provided in this run." - -Choose 1, 2, or 3: -``` - -This prompt blocks before Step 7 provisions a box. Don't burn cost on a verification path the maintainer hasn't agreed to. - -**API-key propagation pattern (for option 1).** Argv exposure is a two-layer problem and the file-based pattern must extend to both layers. - -**Layer 1 — local → Brev (surfaced #2604).** Passing the key as `NVIDIA_API_KEY=<value> brev exec ...` puts the literal value in the brev exec process's argv on the maintainer's laptop *and* on the Brev box (since brev exec serializes argv to the remote shell). Visible in `ps -ef` on both ends for the duration of the run. Use file-based copy: - -```bash -# After Step 6.5 preconditions, copy the local key file to the Brev box. -[ -f ~/.nvidia-api-key ] && brev copy ~/.nvidia-api-key "$INSTANCE_NAME":~/.nvidia-api-key -brev exec "$INSTANCE_NAME" "chmod 600 ~/.nvidia-api-key 2>/dev/null || true" -``` - -**Layer 2 — on-box subshell (surfaced #2611).** Inside scripts running on the Brev box, the outer shell reads the key from `~/.nvidia-api-key` cleanly, but a *naive* inner subshell call leaks it back into argv: - -```bash -# WRONG — the double-quoted outer heredoc interpolates $NVIDIA_API_KEY at -# script-eval time, so the literal nvapi- value lands in `sg docker -c "..."`'s -# argv and shows up in `ps -ef` on the box for the whole onboard window. -NVIDIA_API_KEY=$(cat ~/.nvidia-api-key) -sg docker -c " - export NVIDIA_API_KEY='$NVIDIA_API_KEY' # ← argv leak - nemoclaw onboard ... -" - -# RIGHT — escape the $ so the outer shell does not interpolate, and let the -# inner subshell read the file itself. Argv contains the command string -# `cat ~/.nvidia-api-key`, not the value. -sg docker -c " - export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key) - nemoclaw onboard ... -" -``` - -The same rule applies to any `bash -c "..."`, `bash -lc "..."`, `su -c "..."`, `ssh host "..."`, or other invocation that takes a command string as a single argv element: **never interpolate the key into the string at the outer shell's eval time**. Read the file inside the inner shell so the value lives in env-vars, never in argv. - -Cleanup: when the trap fires `brev delete`, the box (and the key file on it) goes away. On the maintainer's laptop, the file persists until they `rm ~/.nvidia-api-key` — Step 12's session log should remind them. **If the key was previously propagated via cmdline (pre-fix at either layer), treat it as exposed and rotate.** - -**Pure-CLI / pure-sandbox-build bugs are exempt** — those don't actually exercise inference, so the provider doesn't matter even if the issue body mentions one. Heuristic: if Step 6.7's local-first predicate would have fired (no sandbox state, no model server interaction), skip the prompt. - ---- - -## Step 6: Extract the Reproducer - -Extract whatever's available from the issue body. The decision about *whether the reproducer is good enough* lives in Step 8 (validate-on-baseline), not here. - -NV QA files most bugs through an HTML form, so issue bodies are typically a mix of `<pre>...</pre>` blocks and tables — not markdown fenced code blocks. Extraction must handle both shapes. - -1. **Verbatim:** the first markdown fence (```` ``` ```` or ```` ~~~ ````) **or** HTML `<pre>` block containing a `nemoclaw` invocation. Strip surrounding tags and unescape HTML entities before saving to `./reproducer.sh`. No confidence penalty (yet). -2. **No verbatim block found:** leave `./reproducer.sh` absent. Step 8b will synthesize from the issue body on demand and apply the **−30 synth penalty** at that point. - -A robust extractor handles both shapes with the body fetched as JSON. The "anchor word" — what marks a block as a reproducer — must include `nemoclaw`, `openclaw`, AND `openshell`. Issue #2592 surfaced this gap: its reproducer was `openclaw channels add telegram` run inside the sandbox; a `nemoclaw`-only regex would have missed the verbatim block and forced the run through Step 8c synth-repro with a -30 penalty: - -```bash -BODY=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json body -q .body) - -REPRODUCER=$(printf '%s' "$BODY" | python3 -c ' -import re, sys, html -b = sys.stdin.read() -# Anchor word: any of nemoclaw / openclaw / openshell. Issue bodies use whichever -# tool the reporter ran (host-side nemoclaw vs in-sandbox openclaw vs openshell CLI). -ANCHOR = r"(?:nemoclaw|openclaw|openshell)" -m = re.search(rf"```(?:bash|sh)?\n(.*?{ANCHOR}.*?)\n```", b, re.S) -if not m: m = re.search(rf"~~~(?:bash|sh)?\n(.*?{ANCHOR}.*?)\n~~~", b, re.S) -if not m: m = re.search(rf"<pre[^>]*>(.*?{ANCHOR}.*?)</pre>", b, re.S) -if m: - text = re.sub(r"<[^>]+>", "", m.group(1)) - print(html.unescape(text).strip()) -') - -[ -n "$REPRODUCER" ] && printf '%s\n' "$REPRODUCER" > ./reproducer.sh -``` - -The "give up immediately" path is gone. Synthesis happens at validation time so it has the baseline transcript to react to, not just the issue body in isolation. The give-up decision now lands in Step 8c when synth fails to produce a script that actually exposes the bug. - ---- - -## Step 6.5: Verify Preconditions - -Confirm CLI dependencies are available, `brev` is authenticated, and the install URL resolves before paying any cost. Credentials live in `~/.brev/credentials.json` and are reused across shells under the same OS user, so once authenticated the auth check is a no-op until the token expires. - -```bash -# CLI deps — fail fast if anything later in the skill needs them but they're missing. -for cmd in gh brev jq python3 curl; do - command -v "$cmd" >/dev/null 2>&1 || { echo "ERROR: missing required dependency: $cmd"; exit 1; } -done - -# gh identity — every comment posted by Step 10 lands under whatever account `gh` is currently -# authenticated as. Surface that explicitly so the maintainer notices before a public comment -# lands under the wrong handle (this matters when `gh` is multi-token, after a recent re-auth, -# or when running under a service-account hostname). -GH_IDENTITY=$(gh api user --jq .login 2>/dev/null) -if [ -z "$GH_IDENTITY" ]; then - echo "ERROR: gh CLI is not authenticated. Run: gh auth login # then re-run this skill" - exit 1 -fi -echo "gh identity: @$GH_IDENTITY — comments posted by this run will appear under this handle" - -# gh 'project' scope — Step 10 moves fixed-on-latest issues to "Needs Review" on Project 199. Warn if missing. -gh auth status 2>&1 | grep -q "'project'" || echo "[verify-stale] WARN gh missing 'project' scope — Step 10 tracker move will skip. Fix: run 'gh auth refresh -h github.com -s project' in a real terminal." - -# Brev auth — short-circuit only after the auth check, not before. -# When auth fails, give the user a directive recipe (the browser-flow path is -# what works from non-TTY harnesses like Claude Code, not the headless options). -brev ls --json >/dev/null 2>&1 || { - cat <<'MSG' - -ERROR: Brev not authenticated. ~/.brev/credentials.json is missing or the token expired. - -What to do (works from any harness, including non-TTY agent contexts): - - 1. Open a separate Terminal on your laptop. - 2. Run: brev login - A browser opens; complete the auth flow; the CLI exits on success. - 3. Come back here and re-run this skill. Credentials persist to - ~/.brev/credentials.json and every subsequent `brev` call picks them up. - -Headless / no-browser alternatives (when option 1 isn't available): - - brev login --skip-browser # prints a URL, paste into any browser - - brev login --token "$BREV_API_TOKEN" # non-interactive; same env var used - # by test/e2e/brev-e2e.test.ts - -MSG - exit 1 -} - -# Repo labels exist — Step 8.5 / Step 10 can't apply a label that doesn't exist. Check -# canonical label names against the live repo so a mismatch fails fast (issue #2168 hit this: -# spec called the label `wontfix`, but the actual repo label is `status: wont-fix`). -EXPECTED_LABELS=("fixed-on-latest" "verify-inconclusive" "status: wont-fix") -LIVE_LABELS=$(gh label list --repo NVIDIA/NemoClaw --limit 200 --json name --jq '.[].name') -for label in "${EXPECTED_LABELS[@]}"; do - printf '%s\n' "$LIVE_LABELS" | grep -Fxq "$label" || { - echo "ERROR: expected label not on repo: '$label'" - echo " create it with: gh label create '$label' --repo NVIDIA/NemoClaw" - exit 1 - } -done - -# Install URL reachable — fails fast instead of mid-Brev-run if the host is down or the URL changed. -# The default is the public Akamai-hosted entry (301-redirects to the actual installer). The -# `nemoclaw.nvidia.com` host that earlier drafts pointed to is NVIDIA-internal and does not -# resolve from Brev; surfaced during the #2007 e2e run. -INSTALL_URL=${NEMOCLAW_INSTALL_URL:-https://www.nvidia.com/nemoclaw.sh} -curl -fsI "$INSTALL_URL" >/dev/null 2>&1 || { - echo "ERROR: install URL not reachable: $INSTALL_URL" - echo " - Check https://www.nvidia.com/nemoclaw.sh is up (the default Akamai-hosted entry)." - echo " - Override with NEMOCLAW_INSTALL_URL=<alternate-url> if your team mirrors the installer." - echo " - Then re-run this skill." - exit 1 -} -``` - ---- - -## Step 6.7: Try Local Reproduction First - -For pure-CLI reproducers (no sandbox state, no GPU, no integration tokens), try locally before paying for a Brev box. The evidence is identical — `nemoclaw <args>` on a maintainer laptop produces the same exit code and stdout as on a fresh Brev VM, modulo platform differences — and the run is free. - -**Predicate** — local-first applies if **all** of these hold: - -- Reproducer is a sequence of `nemoclaw <args>` invocations only. No `docker`, `kubectl`, `curl`, `npm`, networking setup, or filesystem fixtures. -- Issue has no `Sandbox`-only or `Docker` label and no GPU signal from Step 5. -- `which nemoclaw` resolves on the maintainer's machine and `nemoclaw --version` reports a build at or past `$LATEST` (a build between `$LATEST` and `$LATEST+main` is fine — these only differ by unmerged WIP). -- Maintainer is on Linux or macOS. Windows local repros are out of scope (per Step 3 platform skip rules). - -**If the predicate fires:** - -```bash -LOCAL_VERSION=$(nemoclaw --version 2>&1) -LOCAL_TRANSCRIPT=$(mktemp) -{ time bash reproducer.sh; } >"$LOCAL_TRANSCRIPT" 2>&1 -LOCAL_EXIT=$? -echo "Local: $LOCAL_VERSION, exit $LOCAL_EXIT" -``` - -Compare local result to the issue's "Actual Result" section using the same match rubric Step 8b applies on baseline: - -- **Local matches the issue symptom exactly** (same exit code + same diagnostic output) AND the symptom is the post-fix expected output → skip Brev. Use the local transcript as the verified-on-latest evidence. Step 10's comment must say `Environment: local install (<version>) — Brev provisioning skipped, outcome deterministic from CLI surface alone`. -- **Local result differs from the reported "Actual Result"** → continue to Step 7 and run on Brev. The local environment may be a confound (different OS, dirty config, partial build); remote confirms. -- **Local repro errors out for environmental reasons** (`nemoclaw: command not found`, npm link broken) → continue to Step 7. Treat as inconclusive locally, not a verification failure. - -**If the predicate does not fire:** proceed to Step 7 normally. Most sandbox-touching bugs need Brev. - ---- - -## Steps 7–12 — Execution, Scoring, and Comment - -Once a candidate has cleared Step 6.7's local-first short-circuit and a Brev run is committed to, the rest of the workflow lives in **[reference/execution-and-comment.md](reference/execution-and-comment.md)**: - -- **Step 7** — Reuse or provision the Brev box (concurrency cap, runtime SKU pick, file-based API key copy). -- **Step 8** — Validate the reproducer on baseline, comprehensive reset, install latest, run again. Sub-steps cover dependency bootstrap, brev-exec quirks, synth-repro retry, architectural-drift check, performance and rebuild-cycle bug classes. -- **Step 8.5** — Detect "behavior changed by design" (three signals; short-circuits Brev cost on intentional removals). -- **Step 9** — Score confidence (+50 / +25 / +25 / −30 / −50; cap-at-84 when baseline didn't validate). -- **Step 10** — Compose and post the comment (redaction, 300-word ceiling, three templates, unanswered-question variant when Step 3 sets `UNANSWERED_MAINT_LOGIN`). -- **Step 11** — Infra failure handling (sandbox-build rot is the dominant failure for any version >5–7 patches behind). -- **Step 12** — Log to the activity file. -- **Cadence**, **Out of Scope (v1)**, and the **Companion Behavior** note (release-tag sweep) live there as well. +Verify-stale progress: +- [ ] Select issue(s), latest tag, and reported version +- [ ] Apply skip/idempotency/active-discussion filters +- [ ] Classify environment, provider, and bug class +- [ ] Extract or synthesize a reproducer +- [ ] Verify preconditions and try local-first if eligible +- [ ] If Brev is needed, get plan approval before provisioning +- [ ] Validate the reproducer on baseline, then verify latest +- [ ] Check by-design/static-analysis branch when behavior was removed +- [ ] Score, redact, draft, and self-verify comment links +- [ ] Re-check issue state, post comment/label, update tracker when required +- [ ] Append activity log entry +``` + +## Workflow + +1. **Select candidates and versions.** Read [reference/candidate-selection.md](reference/candidate-selection.md). Use it for single-issue mode, batch mode, latest-tag detection, filters, idempotency, active-discussion handling, and reported-version parsing. +2. **Classify and prepare.** Read [reference/environment-and-reproducer.md](reference/environment-and-reproducer.md). Use it for CPU/GPU/provider/bug-class classification, safe API-key handling, reproducer extraction, dependency checks, Brev auth, label checks, and local-first verification. +3. **Stop for approval before cost.** In batch mode, present one issue's plan and wait for maintainer approval before provisioning Brev. +4. **Provision and install.** If local-first does not settle the issue, read [reference/brev-provisioning.md](reference/brev-provisioning.md). Use it for Brev reuse/provisioning, reset, baseline/latest installs, dependency bootstrap, and `brev exec` footguns. +5. **Run the verification rubric.** Read [reference/reproduction-rubrics.md](reference/reproduction-rubrics.md). Use it to validate baseline behavior, retry with a synthesized reproducer if needed, run latest, handle architectural drift, and branch for performance or rebuild-cycle bugs. +6. **Check intentional changes.** If the symptom targets removed/deprecated behavior, read [reference/by-design.md](reference/by-design.md). Use static evidence to apply `status: wont-fix` only when the by-design branch self-verifies. +7. **Score, comment, label, and log.** Read [reference/scoring-comments-and-logging.md](reference/scoring-comments-and-logging.md). Use it for confidence scoring, redaction, concise templates, issue-state race checks, Project 199 movement, infra failures, and activity logging. + +## Non-negotiables + +- Never auto-close an issue. Apply labels and ask a maintainer/reporter to confirm. +- Never put API keys on a command line. Use the file-based pattern in `environment-and-reproducer.md`. +- Never post unredacted transcripts, issue excerpts, synthesized scripts, internal hostnames, email addresses, or tokens. +- Never post a comment with broken markdown links or tag-drifting `file:line` citations. Re-run cited commands and link-check at least one rendered link per comment section. +- Never use Brev for unsupported platforms or integration-token issues in v1. +- Keep comments concise: default to 200–300 words for fixed/by-design, 100–200 for inconclusive, and 30–80 for still-reproduces. + +## Reference map + +| Need | Read | +|---|---| +| Candidate query, filters, version parser | [reference/candidate-selection.md](reference/candidate-selection.md) | +| Environment classification, credentials, reproducer, preconditions, local-first | [reference/environment-and-reproducer.md](reference/environment-and-reproducer.md) | +| Brev box reuse/provision, reset, installs, dependency bootstrap | [reference/brev-provisioning.md](reference/brev-provisioning.md) | +| Baseline/latest matching, synth-repro, drift, performance, rebuild-cycle | [reference/reproduction-rubrics.md](reference/reproduction-rubrics.md) | +| Static by-design/wont-fix branch | [reference/by-design.md](reference/by-design.md) | +| Score, redact, comment, label, tracker move, infra handling, log | [reference/scoring-comments-and-logging.md](reference/scoring-comments-and-logging.md) | diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/brev-provisioning.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/brev-provisioning.md new file mode 100644 index 00000000000..b6ebe106715 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/brev-provisioning.md @@ -0,0 +1,297 @@ +<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> +<!-- SPDX-License-Identifier: Apache-2.0 --> + +# verify-stale — Brev Provisioning and Install Reference + +Use when the local-first path does not settle the issue and a Brev run is approved. Covers box reuse/provisioning, reset, baseline/latest installs, dependency bootstrap, and `brev exec` footguns. + +## Contents + +- [Step 7: Reuse or Provision a Brev Box](#step-7-reuse-or-provision-a-brev-box) +- [Step 8: Validate on Baseline, Verify on Latest](#step-8-validate-on-baseline-verify-on-latest) +- [Comprehensive reset](#comprehensive-reset-run-before-each-install) +- [Step 8a: Install reported version](#step-8a-install-reported-version) +- [Step 8a.5: Bootstrap reproducer dependencies](#step-8a5-bootstrap-reproducer-dependencies) +- [Step 8a.5b: Brev exec environment quirks](#step-8a5b-brev-exec-environment-quirks) + +--- + +## Step 7: Reuse or Provision a Brev Box + +The skill prefers reuse over provisioning. A pool of `verify-stale-*` boxes (CPU and GPU) can be kept warm; reuse the matching one if available, otherwise provision. + +```bash +# Auth + install URL already verified by Step 6.5 — no need to re-check or auto-login here. + +# Determine class from Step 5: "cpu" or "gpu" +INSTANCE_CLASS="cpu" # or "gpu" + +INSTANCES=$(brev ls --json) + +# Look for an existing running verify-stale-* box matching the required class. +# CPU boxes have no .gpu field set; GPU boxes do. +EXISTING=$(echo "$INSTANCES" | jq -r --arg class "$INSTANCE_CLASS" ' + .[]? + | select(.name | startswith("verify-stale-")) + | select(.status == "RUNNING") + | select(($class == "gpu" and (.gpu // "" != "")) + or ($class == "cpu" and (.gpu // "" == ""))) + | .name' | head -1) + +PROVISIONED_NEW=0 + +if [ -n "$EXISTING" ]; then + INSTANCE_NAME="$EXISTING" + echo "Reusing existing verification box: $INSTANCE_NAME" +else + # Concurrency cap: refuse if 4+ verify-stale-* boxes are already running. + # Filter on .status to match the reuse query above — counting non-running boxes + # would falsely block provisioning when prior boxes are stopped but not deleted. + RUNNING=$(echo "$INSTANCES" | jq '[.[]? | select(.name | startswith("verify-stale-")) | select(.status == "RUNNING")] | length') + if [ "$RUNNING" -ge 4 ]; then + echo "ERROR: 4 verify-stale boxes already running. Wait for one to finish or reuse." + exit 1 + fi + + INSTANCE_NAME="verify-stale-${ISSUE_NUMBER}-$(date +%s)" + + if [ "$INSTANCE_CLASS" = "gpu" ]; then + # brev create auto-selects the cheapest GPU meeting the defaults + # (>=20GB VRAM, >=500GB disk, compute >=8.0). Override with --type if needed. + brev create "$INSTANCE_NAME" + else + # CPU case: pick the cheapest stoppable Linux SKU at runtime so the skill doesn't rot when + # SKUs change. Bias the floor by reproducer-implied memory needs — the cheapest 2 GB SKU + # cannot load a 4.8 GiB Ollama probe, and onboard fails at provider validation before any + # sandbox-creation code runs. Surfaced during the #2007 e2e run (wasted ~25 min on a 2 GB + # box that couldn't load `nemotron-3-nano:4b`). + # + # Memory floor heuristic: + # - Reproducer references Ollama or vLLM or names a model tag (e.g. `nemotron-3-nano:4b`, + # `llama3:8b`) -> floor 16 GB (covers ~5 GB model + sandbox + gateway overhead). + # - Reproducer touches sandbox onboarding without a local model server -> floor 8 GB. + # - Pure CLI-surface bug (no sandbox, no model) -> floor 4 GB. + # Override the auto-pick by exporting VERIFY_STALE_CPU_TYPE if the team has hard preferences. + CPU_RAM_FLOOR=${CPU_RAM_FLOOR:-8} + CPU_TYPE=${VERIFY_STALE_CPU_TYPE:-$(brev search cpu --sort price --json \ + | jq -r --argjson floor "$CPU_RAM_FLOOR" \ + '[.[] | select(.stoppable == true and .ram_gb >= $floor)] | .[0].type')} + [ -n "$CPU_TYPE" ] || { echo "ERROR: no stoppable CPU SKU with >= ${CPU_RAM_FLOOR} GB RAM"; exit 1; } + brev create "$INSTANCE_NAME" --type "$CPU_TYPE" + fi + + PROVISIONED_NEW=1 +fi + +# Cleanup runs on success, error, and SIGINT. +# Delete only what we provisioned. Reused boxes stay warm for next time. +# `brev delete` is non-interactive by default — there is no --yes flag, and passing one errors. +echo ">>> Brev instance: $INSTANCE_NAME (provisioned_new=$PROVISIONED_NEW; manual cleanup: brev delete $INSTANCE_NAME)" +trap '[ "$PROVISIONED_NEW" = "1" ] && brev delete "$INSTANCE_NAME" >/dev/null 2>&1 || true' EXIT +``` + +Wallclock cap per verification: **60 minutes** default. The cap accommodates two full install passes (baseline + latest), comprehensive resets between them, and any reproducer dependency bootstrapping (Step 8a.5) — most of which run sequentially against a single Brev box. Bugs that genuinely require more than an hour to manifest fall out of v1 scope; if a provisioned box isn't ready in time, abort and treat as an infra failure (Step 11). + +The previous design had a 25-min default with a 60-min extension for time-sensitive bugs (`memory leak`, `over time`, etc.). That split optimised for the wrong constraint — most issues fit comfortably under 60 min, and the keyword-based extension forced re-runs whenever a real install or bootstrap took longer than the optimistic 25-min budget. Single 60-min cap removes that paper cut. + +--- + +## Step 8: Validate on Baseline, Verify on Latest + +Two-pass design. + +- **Baseline pass (8a–8c):** install the **reported version**, run the reproducer, confirm it actually exposes the bug as described. This is the gate that proves the script is real. +- **Latest pass (8d):** install **latest**, run the validated reproducer. This is what the confidence score is built on. + +Without the baseline gate, a clean run on latest is ambiguous: maybe the bug really got fixed, maybe the script was never capable of triggering it. The baseline disambiguates. + +### Comprehensive reset (run before each install) + +NemoClaw spawns OpenShell sandboxes (containers), runtime services, and listening processes. A naive `rm -rf ~/.nemoclaw` doesn't clean those — the latest install would inherit baseline state and contaminate the result. Use this fuller reset between installs: + +```bash +RESET=$(cat <<'SCRIPT' +nemoclaw destroy --all --force 2>/dev/null || true +# Anchor pkill patterns to "/nemoclaw" / "/openshell" path components so the kill doesn't +# match unrelated processes that happen to mention these strings (including the agent +# harness running this skill if its working dir contains the word). +pkill -9 -f '/nemoclaw([[:space:]]|$)' 2>/dev/null || true +pkill -9 -f '/openshell([[:space:]]|$)' 2>/dev/null || true +docker ps -a --filter "name=openshell-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true +docker ps -a --filter "name=nemoclaw-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true +# Sandbox state lives in ~/.openclaw (default-writable since #2227); ~/.nemoclaw holds CLI state. +# Wipe both so the latest install starts clean. +rm -rf ~/.nemoclaw ~/.openclaw 2>/dev/null +sudo -n rm -f /usr/local/bin/nemoclaw 2>/dev/null || true +sudo -n rm -rf /usr/local/lib/nemoclaw 2>/dev/null || true +for port in 8080 18789 9119; do fuser -k -n tcp $port 2>/dev/null || true; done +true +SCRIPT +) +``` + +Idempotent — fails silently when there's nothing to clean. Run via `brev exec "$INSTANCE_NAME" "$RESET"` before 8a's install and again before 8d's install. + +**Sudo precondition.** All `sudo` invocations use `sudo -n` (non-interactive) so they fail fast instead of hanging on a password prompt. The skill assumes the Brev image's default user has passwordless sudo configured — Brev's stock images do; custom images may not. If `sudo -n` fails, the binary cleanup is best-effort and a stale `/usr/local/bin/nemoclaw` may persist. The user-local install path (`~/.nemoclaw`) is fully reset regardless. + +### Step 8a: Install reported version + +The installer accepts the target ref via the `NEMOCLAW_INSTALL_TAG` env var (verified against `install.sh` source — defaults to `latest` if unset). It is **not** a `--version` flag. + +```bash +brev exec "$INSTANCE_NAME" "$RESET" + +# Pass the provider env vars through so install.sh's bundled `[3/3] Onboarding` step +# doesn't fall back to the default `build` (NIM) provider — which requires NVIDIA_API_KEY +# and otherwise fails the install with a misleading error. When NEMOCLAW_PROVIDER=ollama +# (the common case), the bundled onboard uses the local Ollama we set up in Step 8a.5 +# and either succeeds (ideal) or fails on a real Dockerfile/sandbox-build issue (which +# is what we want to detect). Pass NVIDIA_API_KEY only if the maintainer provided one +# at Step 5's prompt. +# Read NVIDIA_API_KEY from ~/.nvidia-api-key on the BOX (not from this shell's argv). +# The Step 5 propagation block already brev-copy'd the key file with 600 perms. +brev exec "$INSTANCE_NAME" " + if [ -f ~/.nvidia-api-key ]; then export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key); fi + NEMOCLAW_INSTALL_TAG=$REPORTED_VERSION \ + NEMOCLAW_NON_INTERACTIVE=1 \ + NEMOCLAW_PROVIDER=${NEMOCLAW_PROVIDER:-ollama} \ + NEMOCLAW_MODEL=${NEMOCLAW_MODEL:-nemotron-3-nano:4b} \ + NEMOCLAW_SANDBOX_NAME=verify-stale-install \ + bash -c 'curl -fsSL $INSTALL_URL | bash' +" || BASELINE_INSTALL_FAILED=1 + +# Verify the resolved install version matches the requested version. This guards against the +# `VAR=val curl ... | bash` shell-scoping footgun where the env var binds to curl, not the +# downstream bash, and the install silently falls through to "latest". Surfaced during a +# rot-debugging investigation where v0.0.36 was silently installed when v0.0.26 was requested +# and several minutes of "convincing" output ran before anyone noticed. Always print the +# resolved state, never trust the requested state. +RESOLVED=$(brev exec "$INSTANCE_NAME" "bash -lc 'nemoclaw --version'" 2>&1 | tail -1) +echo "[verify-stale] baseline requested: $REPORTED_VERSION; resolved: $RESOLVED" +case "$RESOLVED" in + *"$REPORTED_VERSION"*) ;; # match — proceed + *) + echo "ERROR: baseline install resolved to '$RESOLVED' but $REPORTED_VERSION was requested." + echo " Common cause: env-var scoping in the install command. Verify the env vars are on" + echo " the BASH side of the curl|bash pipe, not the curl side. Setting" + echo " BASELINE_INSTALL_FAILED=1 to prevent verifying against the wrong version." + BASELINE_INSTALL_FAILED=1 + ;; +esac + +# The bundled onboard creates a sandbox name we don't want carrying through to the reproducer. +# Use a hyphen-only name (NemoClaw's name validator rejects underscores). Destroy it so the +# reproducer starts from a clean state. +brev exec "$INSTANCE_NAME" "sg docker -c 'nemoclaw destroy --all --force 2>/dev/null || true'" +``` + +If install fails (old releases rot — installer URLs, deps, OS images all drift over time, or the in-image Dockerfile patch step asserts against a code shape that's since changed), set `BASELINE_INSTALL_FAILED=1` and **skip 8b/8c**, going straight to 8d. Note "baseline-install-skipped" or "baseline-build-skipped" in the final comment depending on which phase rotted. Step 9's scoring rule handles the degraded mode (cap at 84). + +**The reproducer's own `nemoclaw onboard` (Step 8b) must pass `--fresh`.** If install.sh's bundled onboard was in an in-progress or failed state when we destroyed the install sandbox, the reproducer's onboard would error with `Previous onboarding session failed. Re-run with --fresh to discard it`. `--fresh` ensures a clean start. + +### Step 8a.5: Bootstrap reproducer dependencies + +Brev's stock CPU images ship with NemoClaw installable but not the broader ecosystem the reproducer may need — local model servers (Ollama, vLLM), inference providers, third-party CLIs. **Default to maximum faithfulness: install the actual dependency the reporter used rather than substituting a stub.** Substituting trades faithfulness for speed; that trade is rarely worth it on a 60-min budget, and it almost always introduces a confound that makes the verdict less trustworthy. + +**When to bootstrap (not substitute):** + +- The reproducer references a specific model/server runtime (`NEMOCLAW_PROVIDER=ollama`, `NEMOCLAW_PROVIDER=vllm`, etc.). +- The reproducer references a specific model name with a tag (`nemotron-3-nano:4b`, `llama3:8b`, etc.). +- The reporter's environment in the issue body shows a configured provider (e.g., `OpenShell CLI: 0.0.26` plus an Ollama running on host). + +**When to substitute (with -30 penalty):** + +- Provider requires an API key the skill cannot safely supply (NIM, OpenAI, Anthropic, etc.). Stubbing a key won't pass validation faithfully and a real key shouldn't sit in a verify-stale run. Apply the -30 penalty (treat as synth-repro per Step 8b) and document the substitution in the comment. +- The bug is *provably* independent of the dependency (e.g., a CLI argument-parsing bug that errors before any provider runs). Note this explicitly in the comment. + +**Canonical bootstraps:** + +```bash +# Ollama + a specific model. +# The Ollama installer registers a systemd service (`ollama.service`) so the +# daemon survives between brev exec calls. +brev exec "$INSTANCE_NAME" "curl -fsSL https://ollama.com/install.sh | sh" +brev exec "$INSTANCE_NAME" "sudo systemctl start ollama && sleep 3" +brev exec "$INSTANCE_NAME" "ollama pull <model>" +brev exec "$INSTANCE_NAME" "ollama list" # confirm before continuing +``` + +```bash +# vLLM + a model (HuggingFace-hosted). +brev exec "$INSTANCE_NAME" "pip install --quiet vllm" +brev exec "$INSTANCE_NAME" "nohup python -m vllm.entrypoints.openai.api_server --model <model> --host 127.0.0.1 --port 8000 >/var/log/vllm.log 2>&1 &" +brev exec "$INSTANCE_NAME" "sleep 30 && curl -fsS http://127.0.0.1:8000/v1/models" +``` + +Bootstrap **once before Step 8b's baseline run** and reuse for Step 8d's latest run. Don't reset Ollama/vLLM state between baseline and latest in the comprehensive reset — model downloads are expensive and unrelated to the NemoClaw install. Adjust the reset script to skip these external services explicitly if needed. + +**If bootstrap fails** (network issue pulling the model, service won't start, etc.), this is an infra failure — abort to Step 11. Do not silently substitute; the user opted into faithfulness for a reason. + +**Ollama coverage table.** Ollama is the default provider for verification runs because it's free, local, and self-hosted. It covers most bug classes faithfully but not all. Use this table to decide whether Ollama is sufficient or whether Step 5's API-key prompt should fire: + +| Bug class | Ollama covers? | Notes | +|---|---|---| +| CLI surface (subcommand parsing, flag handling, oclif dispatch) | ✓ Always | Provider not exercised | +| Sandbox structure (build, file permissions, mounts, layout) | ✓ Always | Provider not exercised | +| Networking / policy (port forwards, NAT, egress rules, channels guards) | ✓ Always | Provider not exercised | +| Generic inference flow (does an agent turn complete, does the proxy route correctly) | ✓ Usually | Ollama can fail in the same shape as NIM/Gemini for most flow bugs | +| Provider-specific behavior (`Provider: NVIDIA` symptom, NIM-only error handling, `Provider: Gemini` quirks) | ✗ No | Different code paths; substitution doesn't exercise the bug | +| Model-specific behavior (`gemini-flash-3-preview` doesn't handle prompt X, `nemotron-3-nano:4b` works fine) | ✗ No | Wrong model = wrong outputs | +| Ollama-shape-specific (#2519 "Ollama-local 401" — local-vs-networked Ollama config) | △ Sometimes | A generic Ollama install may or may not reproduce; may need specific configuration | +| Performance / latency on specific silicon | ✗ No | Hardware substitution caveat (Step 10) and Step 8e perf rubric apply | +| Quota / rate-limit / API-key validation | ✗ No | Ollama doesn't have those failure modes | + +When the table says ✗ No or △ Sometimes, Step 5's API-key prompt fires. When it says ✓, proceed with Ollama and skip the prompt. + +### Step 8a.5b: Brev exec environment quirks + +Two non-obvious gotchas surfaced during the #2007 e2e run that every subsequent `brev exec` call has to handle. Encode them once here so reproducer scripts don't have to relearn each time. + +**PATH does not include `~/.local/bin` in non-login shells.** `nemoclaw`'s installer drops a shim at `~/.local/bin/nemoclaw` and updates PATH via `~/.bashrc` / `~/.profile`. `brev exec` spawns non-login, non-interactive shells that don't source those files, so a bare `brev exec "$INSTANCE" "nemoclaw --version"` returns `command not found` on a freshly-installed box. Fix: every reproducer script must explicitly export PATH at the top, OR every `brev exec` call must wrap with `bash -lc '...'`. + +```bash +# Reproducer scripts: prepend this line. +export PATH="$HOME/.local/bin:$PATH" + +# Or equivalently when calling brev exec ad-hoc: +brev exec "$INSTANCE" "bash -lc 'nemoclaw --version'" +``` + +**Docker group requires `sg docker -c '...'` after `usermod -aG`.** Adding the user to the `docker` group (`sudo usermod -aG docker ubuntu`) takes effect for new login sessions, but `brev exec` calls in the same Brev session keep the old gid. The reproducer's `nemoclaw onboard` will fail with `permission denied while connecting to /var/run/docker.sock` unless the call runs in a subshell with the docker group active. + +```bash +# Reproducer execution: wrap with sg docker. +brev exec "$INSTANCE" "sg docker -c 'bash ~/reproducer.sh'" +``` + +Both patterns appear in the canonical setup script committed alongside the skill (or are encoded in your reproducer wrapper). Don't rely on the user discovering them mid-run. + +**`openshell sandbox exec` argument-order footgun.** When the reproducer needs to run a command *inside* the sandbox (channels-guard checks, in-sandbox file inspection, etc.), the correct non-interactive form uses `-n <name>` and a `--` separator: + +```bash +# Correct: +openshell sandbox exec -n ai -- bash -c 'source /sandbox/.bashrc; openclaw channels add telegram; echo "EXIT=$?"' + +# Wrong (silently auto-detects sandbox by "last used", stuffs the leftover positional +# `ai` into bash's $0, prints "/bin/bash: line 1: ai: command not found" — the +# reproducer appears to fail but actually never ran inside the sandbox at all): +openshell sandbox exec ai bash -c '...' +``` + +Issue #2592's first run hit this — wasted ~15 min before the maintainer noticed. Always use the `-n <name> -- <cmd>` form when the reproducer touches in-sandbox commands. + +**`brev exec` SSH-drop re-execution guard.** Brev's CLI silently retries from the top when the SSH connection drops mid-run, producing two parallel reproducer executions (we hit this on #2592 — one onboard process clobbered another's state, and both got billed). Use a sentinel file in the reproducer wrapper to make the script idempotent: + +```bash +# At the top of the reproducer wrapper script: +SENTINEL=~/.verify-stale-running +if [ -f "$SENTINEL" ]; then + echo "ERROR: another verify-stale run is in progress (sentinel: $SENTINEL)." + echo " If you're sure no other run is active, rm $SENTINEL and re-invoke." + exit 1 +fi +trap 'rm -f "$SENTINEL"' EXIT +touch "$SENTINEL" +``` + +The sentinel survives an SSH drop because it lives on the Brev box's filesystem; the trap removes it on script exit. A second `brev exec` invocation that tries to retry from the top will hit the sentinel and bail instead of double-running. diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/by-design.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/by-design.md new file mode 100644 index 00000000000..50c4777f8fd --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/by-design.md @@ -0,0 +1,178 @@ +<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> +<!-- SPDX-License-Identifier: Apache-2.0 --> + +# verify-stale — By-Design Detection Reference + +Use whenever the reproducer points at removed, intentionally changed, or deprecated behavior. This branch can short-circuit Brev cost and label `status: wont-fix`, but every claim needs verifiable evidence. + +## Contents + +- [Step 8.5: Detect "Behavior Changed by Design"](#step-85-detect-behavior-changed-by-design) +- [Signal detection](#step-85a-run-signal-detection) +- [Related failure modes](#step-85b-pre-check-related-failure-modes) +- [Existing test coverage](#step-85c-check-existing-test-coverage) +- [Self-verification pass](#step-85d-self-verification-pass-before-posting) +- [By-design comment template](#by-design-comment-template) + +--- + +## Step 8.5: Detect "Behavior Changed by Design" + +Before scoring, check whether the symptom is intentional. Some bugs are filed against behavior that was **deliberately changed or removed** in a merged PR — running the standard rubric on these produces misleading verdicts. The symptom "still reproduces" but the right answer is "won't fix, see PR #X." Issue #2791 is the prototype: `config set` was removed in PR #2227, the reporter tested a version that already had it gone, and a standard rubric run would have buried that context under a low-confidence `verify-inconclusive` label. + +This step is split into substeps so the rigor is mechanical, not optional. Every claim in the final comment must be backed by a verifiable evidence block — a comment URL with quoted phrase, a commit SHA with diff range, or a grep command with its actual output. Hand-wavy claims fail Step 8.5d's self-verification pass and force a bail to `verify-inconclusive`. + +### Step 8.5a: Run signal detection + +Any single signal is sufficient to trigger the by-design branch. + +**Signal 1 — Maintainer attribution in comments.** Any comment by an author with `authorAssociation` of `MEMBER`, `OWNER`, or `COLLABORATOR` matches `removed in #\d+`, `removed in [Pp][Rr] ?#\d+`, `by design`, `wontfix`, `won't fix`, `not a bug`, or `intentional`. + +```bash +gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ + --jq '.comments[] + | select(.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR") + | select(.body | test("removed in #\\d+|by design|wontfix|won.t fix|not a bug|intentional"; "i")) + | {url, author: .author.login, body}' +``` + +Capture for evidence: comment URL + author login + the exact quoted phrase. + +**Signal 2 — Removal commit in range.** A commit between the reported version and `$LATEST` deletes the symbol implicated by the reproducer (CLI subcommand, function, flag). The commit subject does NOT need to mention "remove" / "delete" — many removals ride into a `refactor(...)` or `feat(...)` commit (e.g. PR #2227 removed `--dangerously-skip-permissions` under a `refactor(sandbox): ...` subject). Use git's pickaxe to find the responsible commit by content: + +```bash +# Pickaxe: list every commit whose diff changes the count of <symbol> occurrences. +# Reverse order so the earliest removal commit lands first in the list. +git log "$REPORTED_VERSION".."$LATEST" -S'<symbol>' --reverse --oneline -- src/ bin/ nemoclaw/src/ + +# Subject-keyword narrowing is only a SUPPLEMENTARY lookup — useful when the +# pickaxe returns many commits and you want to focus on the obviously-removal one. +git log "$REPORTED_VERSION".."$LATEST" --grep='remove\|delete\|drop\|deprecate' -i --oneline + +# For each candidate, confirm the diff actually deletes the symbol (not just renames or moves it). +git log -p <candidate-sha> -- src/ bin/ nemoclaw/src/ | grep -nE '^-.*\b<symbol>\b' +``` + +Capture for evidence: commit SHA + each `file:line` block of deletions touching the symbol. Note the commit's actual subject — don't assume it says "remove." + +**Signal 3 — Symbol absent in both reported version and latest.** The implicated symbol (e.g. `config set`) is not present in either tag's source tree — meaning the responsible change landed before the version the reporter tested. This is the #2791 case. + +```bash +git grep -n "<symbol>" "$REPORTED_VERSION" -- src/ bin/ nemoclaw/ # expect: zero matches (or shim-only — see sub-case) +git grep -n "<symbol>" "$LATEST" -- src/ bin/ nemoclaw/ # expect: zero matches (or shim-only) +``` + +Capture for evidence: both grep commands and their (empty) outputs. + +**Sub-case for signals 2 and 3 — vestigial deprecation shims.** It's common for a removed symbol to survive in latest *only* as a deprecation message (e.g., a CLI subcommand that prints `"--<flag> was removed; use <X> instead"` and exits non-zero). When a grep returns matches in latest, inspect each `file:line`. If every match is a deprecation stub with no functional effect on the bug-as-filed, signal 2 or 3 still fires; record the shim locations and behavior as a separate evidence block. Do not silently treat shims as functional code, and do not silently treat them as absence. + +### Step 8.5b: Pre-check related failure modes + +A by-design verdict says "the bug *as filed* can't reproduce." It does NOT say "every bug shaped like this is fixed." Before drafting the comment, search latest's source for code paths that could still produce the issue's described **symptom** (not the literal removed flag/symbol — the symptom). + +```bash +# Use the issue's symptom keywords, not the removed symbol. +git grep -nE "<symptom-keyword-1>|<symptom-keyword-2>" "$LATEST" -- src/ nemoclaw/src/ +``` + +For #2168 the literal flag is `--dangerously-skip-permissions`, but the symptom is "sandbox created but not registered in CLI." Grepping for `register.*[Ss]andbox`, the readiness-gate / cleanup-failure path in `src/lib/onboard.ts` surfaces as a related-but-different way to produce an orphan sandbox. + +If a related failure mode is found, the by-design comment MUST include a "What's not literally the same bug" section that names it with `file:line`. Don't suppress the call-out by claiming "the symptom is impossible" when the symptom can be reached via a different path. + +### Step 8.5c: Check existing test coverage + +Search the repo for tests that exercise the NEW intended workflow (the one that replaced the removed symbol). Citing them strengthens the comment from "trust me, it was removed" to "the new workflow is exercised by these tests." + +```bash +git grep -lnE "<new-workflow-keyword>" -- test/ nemoclaw/src/ 2>/dev/null | head -5 +``` + +Cite at most three concrete test paths. If none exist, omit the section — do not invent paths. + +### Step 8.5d: Self-verification pass before posting + +Two passes, both required. + +**Evidence pass.** Re-run every grep / git / `gh` command cited in the evidence blocks. If any cited `file:line`, commit SHA, or quoted output doesn't reproduce on a fresh invocation, **stop and revise** — or bail to `verify-inconclusive` if the discrepancy can't be resolved. + +**Link pass.** Resolve at least one rendered markdown link from each section that has them — `What's structurally fixed`, `Vestigial references`, `Existing CI coverage`. Use `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 content if the path exists at the tag, 404 otherwise) or `curl -fsI <blob-url>` (returns 200 if the blob renders). A broken link is worse than no link — it suggests verification work that didn't actually happen. + +The cost of an incorrect "I checked and X is gone" claim in a public comment, or a 404 on a citation, is higher than spending a minute re-checking. This step exists because LLMs can confidently overstate and confidently invent paths; mechanical re-verification catches both. + +### Step 8.5e: If any signal fires + +- **Skip the Step 9 score table** entirely. The "exit 0 + expected output" axis doesn't apply when the expected output is no longer the contract. +- **Skip Brev provisioning** if the signal fires before Step 7 — a remote run would just confirm what static analysis already proved. (Signals 2 and 3 can run as soon as the reported version is parsed in Step 4.) +- **Apply label `status: wont-fix`** (the existing repo label — quote it on the CLI: `gh issue edit <num> --add-label "status: wont-fix"`). It's already in the Step 3 issue-type skip list, so a labelled issue is automatically excluded from future runs without needing a separate idempotency clause. +- **Use the by-design comment template below** instead of the standard Step 10 template. +- **@-mention the reporter** so they can object if the framing is wrong. +- **Never auto-close.** A maintainer pulls the trigger, same as the other label paths. + +### By-design comment template + +Mandatory sections in this order. Omit only the sections explicitly noted as omittable. + +**Tag-anchoring + linking rule.** Every `file:line` citation, commit SHA, and test-path reference in the rendered comment MUST be a clickable markdown link to the verified-on tag (e.g., `v0.0.35`), not the maintainer's working `HEAD`. Lines drift between tags and main; tag-anchored links keep the citations reproducible by anyone reading the comment months later. Bare paths force the reader to navigate manually — that's a usability bug, not a stylistic preference. + +Use these exact link formats: + +- File only: `[src/lib/onboard.ts](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/onboard.ts)` +- File:line: `[src/lib/onboard.ts:4965](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/onboard.ts#L4965)` +- File:line-range: `[src/lib/commands/sandbox/connect.ts:25-31](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/commands/sandbox/connect.ts#L25-L31)` +- Commit SHA: `[5956a61](https://github.com/NVIDIA/NemoClaw/commit/5956a612e18047b9ab85b3a7e89f6b5dedb29190)` — short SHA as the link text, full SHA in the URL +- Test file: `[test/e2e/test-double-onboard.sh](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/test/e2e/test-double-onboard.sh)` +- PR/issue references: bare `#NNNN` works — GitHub auto-links these in comments on the same repo, no manual URL needed. + +When greping for evidence, use `git grep -n "<symbol>" "$LATEST" -- ...` so the line numbers match the tagged blob. Then construct each link from `<file path> + verified-on tag + line number`. + +The Step 8.5d self-verification pass MUST resolve at least one rendered link (e.g., `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=v0.0.35` or a `curl -fsI` to the blob URL) and confirm it returns the expected file. A broken link defeats the purpose of including the citation. If any link fails to resolve, fix it or bail to `verify-inconclusive`. + +````markdown +## Stale-issue verification — behavior is by-design + +**Reported on:** v0.0.<X> +**Verified on:** v0.0.<Y> (PR #<NNNN> first shipped in v0.0.<Z>) +**Verification mode:** static analysis at the verified-on tag — no runtime reproduction. Step 8.5 by-design short-circuits Brev provisioning because the responsible code change is already proven by the diff between `$REPORTED_VERSION` and `$LATEST`. +**Outcome:** symptom reproduces against the reproducer as filed, but the implicated behavior was intentionally changed. + +### What's structurally fixed + +- `<file:line>` — `<one-sentence summary of the change at that location>` +- `<file:line>` — `<…>` + +The new workflow is `<one-sentence: how to do what the user was trying to do>`. + +### Vestigial references + +- `<file:line>` — `<deprecation behavior: e.g. "prints '--<flag> was removed; use <X> instead' and exits 1; no functional effect">` + +(Omit this section entirely when the symbol is fully gone with no surviving stubs.) + +### What's not literally the same bug + +`<one-sentence acknowledgement of the related failure mode found in Step 8.5b, with file:line>` — OR — `None. The symptom requires the removed symbol; no related code path produces it on latest.` + +### Existing CI coverage + +- `<test/path/file>` — `<one-sentence: what this test demonstrates about the new workflow>` + +(Omit when no direct test exists. Do not invent paths.) + +### Recommendation + +@<reporter> — please confirm the by-design framing is correct (the implicated `<symbol>` was intentionally removed, the original reproducer can no longer execute) and close as "won't fix / by design" if you agree. If a related symptom (e.g. `<related failure mode from above>`) is hitting you on ≥ v0.0.<Z>, please file a fresh issue with a v0.0.<Z>+ reproducer. + +`<NVBugs cross-ref line — see below>` + +<!-- nemoclaw-verify-stale v1 YYYY-MM-DD --> +```` + +**NVBugs cross-ref line.** If `NVBUGS_REF` was set in Step 4, append: + +> NVBugs<NVBUGS_REF without brackets> will need a separate update; closing this GitHub issue won't propagate. + +Otherwise omit the sentence. + +**If no signal fires:** continue to Step 9 normally. + +--- diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md new file mode 100644 index 00000000000..9594c3674ca --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md @@ -0,0 +1,238 @@ +<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> +<!-- SPDX-License-Identifier: Apache-2.0 --> + +# verify-stale — Candidate Selection Reference + +Use after loading `SKILL.md` to choose an issue and establish its reported NemoClaw version. + +## Contents + +- [Step 1: Determine Mode](#step-1-determine-mode) +- [Step 2: Detect the Latest NemoClaw Version](#step-2-detect-the-latest-nemoclaw-version) +- [Step 3: Filter Candidates](#step-3-filter-candidates) +- [Step 4: Parse Reported Version](#step-4-parse-reported-version) + +--- + +## Step 1: Determine Mode + +**Single-issue mode** — user provides an issue number: + +```bash +gh issue view <number> --repo NVIDIA/NemoClaw \ + --json number,title,body,labels,url,author,createdAt,comments +``` + +**Batch mode** — user says "batch", "weekly", or provides no number. Cap at **15 issues** for *processing* per run, enforced as a slice after Step 3/4 filters narrow the pool. The cap exists because batch is sequential (Step 7 reuse-or-provision keeps it on 1–2 Brev boxes total) and the wallclock budget is ~2–3 hours per 15-issue run; running larger forces the maintainer to either drop the per-plan approval gate or spread the batch across multiple sessions. + +The discovery query needs to see the entire open-bug pool — the per-run processing cap is downstream. Use `--limit 1000` so the skill doesn't silently drop issues beyond the page (the candidate triage run found 129 open bugs; an earlier `--limit 100` would have missed 29 of them). + +```bash +gh issue list --repo NVIDIA/NemoClaw --state open --limit 1000 \ + --label bug \ + --json number,title,body,labels,url,author,createdAt,comments +``` + +In batch mode, work through items one at a time. Present each verification plan and wait for approval before any Brev provisioning. + +--- + +## Step 2: Detect the Latest NemoClaw Version + +Try GitHub releases first; fall back to the highest semver tag from the GitHub API if no release is published. NemoClaw currently tags but does not publish releases, so the fallback is the load-bearing path today. Use `gh api` rather than `git ls-remote` so the skill works regardless of SSH key setup, and reuses the auth `gh` already has. + +```bash +LATEST=$(gh release view --repo NVIDIA/NemoClaw --json tagName -q .tagName 2>/dev/null) + +if [ -z "$LATEST" ]; then + LATEST=$(gh api repos/NVIDIA/NemoClaw/tags --paginate --jq '.[].name' \ + | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' \ + | sort -V | tail -1) +fi + +echo "Latest tag: $LATEST" +``` + +This is the version the skill will verify against. Record it — every comment must cite it. + +--- + +## Step 3: Filter Candidates + +Apply these rules in order. Drop any issue that fails a rule. + +**Issue-type allowlist:** must have `bug` label. +**Issue-type skip:** drop if any label exactly matches `documentation`, `status: wont-fix`, `status: needs-info`, `security`, OR is `enhancement` / starts with the prefix `enhancement:` (the repo has 8 prefixed variants — `enhancement: feature`, `enhancement: MCP`, `enhancement: testing`, `enhancement: ui`, `enhancement: provider`, `enhancement: platform`, `enhancement: policy`, `enhancement: inference`, `enhancement: integration`, `enhancement: performance`, `enhancement: skill` — and exact-match misses them all; surfaced from #1752). Use the canonical repo label names — bare `wontfix` / `needs-info` are NOT the repo's labels (verified via `gh label list`); the actual labels carry a `status:` prefix and a hyphen. + +**Platform skip (Brev-reproducible only in v1):** drop if any of `Platform: Windows/WSL`, `Platform: MacOS`, `Platform: macOS`, `Platform: Jetson AGX Thor/Orin`. Brev has no equivalent hardware for Jetson (embedded/edge ARM with integrated GPU is not in the Brev SKU catalog), so any Brev verification of a Jetson-only bug would produce a misleading "fixed-on-x86" verdict. Keep `Platform: Ubuntu`, `Platform: DGX Spark`, `Platform: GB10`, `Platform: All`, or no platform label. `Platform: DGX Spark` and `Platform: GB10` stay in scope but Step 10 requires a "Hardware substitution" caveat in the comment naming the Brev SKU we used as a substitute (Brev x86 GPU SKUs are not faithful to GB10 / Grace Hopper silicon for performance-shape or memory-architecture-shape bugs). + +**TUI / interactive-UI skip:** drop if the issue title contains `TUI`, `dashboard UI`, `chat UI`, `keystroke`, or `key press`, OR if the body describes interactive UI behavior (key sequences, mouse interactions, browser-side UI state) without a non-interactive reproducer (no `NEMOCLAW_NON_INTERACTIVE=1` or equivalent env var pattern). `brev exec` does not allocate a real TTY by default, so TUI reproducers hang or silently fail at the first prompt; v1 documents this as out-of-scope rather than emitting a wrong verdict. v1.1 may add a `script(1)` / `expect` / `tmux send-keys` harness to lift this skip. + +**Integration skip (deferred to v2):** drop if any of `Integration: Slack`, `Integration: Discord`, `Integration: Telegram`, `Integration: Hermes`, `Integration: OpenClaw`, `Integration: WeChat`. These need third-party credentials a fresh Brev box cannot provide. + +**Component allowlist (must have at least one):** `NemoClaw CLI`, `Sandbox`, `OpenShell`, `Docker`, `Getting Started`, or any `Platform:` label that survived the platform skip. + +**Idempotency:** drop if **either** of these is true: + +- The issue carries a `fixed-on-latest` or `verify-inconclusive` label. (Cleared by the release sweep in `nemoclaw-maintainer-cut-release-tag` so the issue re-opens on each release.) The by-design path uses the existing repo `status: wont-fix` label, which is already covered by the issue-type skip rule above — no separate idempotency clause needed for that path. +- A comment matching `<!-- nemoclaw-verify-stale v\d+ YYYY-MM-DD -->` was posted **within the last 7 days**. The regex matches any marker version (`v1`, `v2`, …) so future skill versions can re-verify older-marked issues by tightening the regex (e.g. require a specific marker version). The marker carries a date so the candidate filter can apply a TTL — useful for the still-reproduces case (Step 9), where no label is applied and we want next week's run to re-verify rather than skip forever. + +Implementation — match the marker against each comment's `createdAt`. Use `gh issue view --json comments` (single-issue mode already fetches this; batch mode's `gh issue list` also returns the comment array per issue): + +```bash +# Cutoff for the 7-day TTL. macOS and Linux date(1) syntax differ; try both. +SEVEN_DAYS_AGO=$(date -u -v-7d +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \ + || date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ) + +# Returns the timestamp of the most recent marker comment within the TTL, or empty. +RECENT_MARKER=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ + --jq --arg cutoff "$SEVEN_DAYS_AGO" ' + .comments[] + | select(.body | test("<!-- nemoclaw-verify-stale v\\d+ \\d{4}-\\d{2}-\\d{2} -->")) + | select(.createdAt > $cutoff) + | .createdAt' \ + | head -1) + +if [ -n "$RECENT_MARKER" ]; then + echo "Skip: marker posted $RECENT_MARKER (within 7-day TTL)" + # In single-issue mode: exit 0 with a friendly message. + # In batch mode: continue to the next candidate. +fi +``` + +Run this check for every candidate that survived the label-based filters above; drop those whose `RECENT_MARKER` is non-empty. + +**Unanswered-maintainer-question handling.** Find the most recent maintainer (`MEMBER`, `OWNER`, `COLLABORATOR`) comment that **looks like a question** (`?`, polite imperative like "please confirm/share/clarify", or starter like "could you / can you / do you") AND that the reporter has not replied to since. Pure triage acknowledgments (`"✨ Thanks for reporting…"`) are skipped. The age of the qualifying comment determines skip-or-proceed: + +- **Within 7 days:** **skip the issue** — the discussion is active, the skill running on top would conflict with the maintainer's framing or confuse the reporter. Surfaced during pre-flight on #2757; running verify-stale on top of a fresh "let me clarify what you observed" question from @cjagwani would have stomped on that conversation. +- **Older than 7 days:** **proceed with verification, but use the unanswered-question comment variant.** After 7 days the maintainer's question has either been forgotten or the reporter has dropped the ball; an independent skill verdict becomes the *unsticking voice* rather than a clueless interruption. The comment leads with a markdown link to the maintainer's unanswered comment (use the unanswered-question shape from the comment templates) and @-mentions BOTH the maintainer and the reporter, not just the reporter. Reuse `$SEVEN_DAYS_AGO` from the marker-TTL check above. + +```bash +REPORTER=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json author --jq .author.login) + +# Most recent unanswered maintainer comment that looks like a question — filters out triage acknowledgments (#1642 surfaced this). +UNANSWERED_MAINT=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ + --jq --arg reporter "$REPORTER" --arg cutoff "$SEVEN_DAYS_AGO" ' + (.comments + | map(select((.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR") + and (.body | test("\\?|(?i)\\bplease (confirm|share|provide|clarify|tell|verify|check|let me know|let us know)|(?i)\\b(could|can|would) you\\b|(?i)\\bdo you (have|know|see|use)\\b")))) + | sort_by(.createdAt) | last) as $maint + | if $maint == null then null + else + ((.comments + | map(select(.author.login == $reporter and .createdAt > $maint.createdAt)) + | length) as $replies + | if $replies > 0 then null + else { + createdAt: $maint.createdAt, + url: $maint.url, + login: $maint.author.login, + recent: ($maint.createdAt > $cutoff) + } + end) + end') + +if [ -n "$UNANSWERED_MAINT" ] && [ "$UNANSWERED_MAINT" != "null" ]; then + MAINT_RECENT=$(printf '%s' "$UNANSWERED_MAINT" | jq -r .recent) + MAINT_DATE=$(printf '%s' "$UNANSWERED_MAINT" | jq -r .createdAt) + MAINT_LOGIN=$(printf '%s' "$UNANSWERED_MAINT" | jq -r .login) + MAINT_URL=$(printf '%s' "$UNANSWERED_MAINT" | jq -r .url) + + if [ "$MAINT_RECENT" = "true" ]; then + echo "Skip: active maintainer discussion (unanswered comment from @$MAINT_LOGIN at $MAINT_DATE, within 7 days)" + # Single-issue mode: exit 0 with the message; batch mode: continue to next candidate. + else + echo "[verify-stale] proceeding with unanswered-question variant — @$MAINT_LOGIN's comment from $MAINT_DATE is older than 7 days" + # Step 10's comment template will lead with the unanswered-question prefix and @-mention + # both the maintainer and the reporter. Export these for the templater: + export UNANSWERED_MAINT_LOGIN="$MAINT_LOGIN" + export UNANSWERED_MAINT_URL="$MAINT_URL" + export UNANSWERED_MAINT_DATE="$MAINT_DATE" + fi +fi +``` + +When the unanswered-question variant fires (`UNANSWERED_MAINT_LOGIN` set), Step 10's comment template prepends a lead paragraph (exact shape lives with the templates in Step 10), and the closing @-mention block names BOTH the maintainer (acknowledging their question) and the reporter (asking for confirmation per the standard pattern), instead of just the reporter. + +**Candidate rule:** keep the issue if **either**: + +- The reported version (parsed from body or labels — see Step 4) is **at least 2 versions behind** `$LATEST` in the rightmost-incrementing component, **or** +- The issue is **older than 7 days** AND a specific version is parseable from its body or labels. + +For NemoClaw's current `0.0.x` line, "rightmost-incrementing component" is the patch number — a v0.0.31 report against a v0.0.34 latest is 3 versions behind. Once NemoClaw moves to `0.1.x` or higher, the rule applies to the next-rightmost component instead. Pick whichever component is actively iterating. + +--- + +## Step 4: Parse Reported Version + +The regex is intentionally **release-line agnostic**. Today NemoClaw ships `v0.0.x`, but the same parser must keep working when it moves to `v0.1.x`, `v1.x.x`, or anything else. Don't hardcode the major/minor digits. + +Sources, in order of trust: + +1. **Labels.** Any label that exactly matches `^v\d+\.\d+\.\d+$` AND appears in the repo's tag list. Labels matching the regex but absent from tags (e.g. `v0.0.35` as a *release-target* milestone before that version ships) are roadmap markers, not "reported on" — drop them. +2. **Body.** Use a **proximity-anchored** regex: `(?i)nemoclaw[^a-z\n]{0,80}v?(\d+\.\d+\.\d+)`. This matches a version that follows `nemoclaw` within 80 non-letter, non-newline characters, capturing just the semver. The anchoring is load-bearing — without it the parser also picks up `openshell 0.0.4`, Node.js `v22.16.0`, IP addresses (`0.0.0.0:11434`, `127.0.0.1`), and other near-NemoClaw products that happen to share the `v0.0.x` line. (This was confirmed in the dry-run: a non-anchored parser produced 12 false-positive candidates whose smallest tag-valid version was actually OpenShell's, not NemoClaw's.) +3. **Comments by the original reporter** — same anchored regex as the body. + +Collect every match from sources 2 and 3 (a single body may mention multiple versions — `0.0.6 and v0.0.10`). Then validate. + +**Validate against the tag list.** A parsed version must exist as a real git tag, otherwise drop it. This single check kills four classes of error in one pass: + +- Reporter typos that cite a non-existent version (`v0.1.0` when only `v0.0.x` is released — observed 3× in the live backlog). +- Calver mistakes (`2026.3.11` — observed 1×). +- Future roadmap labels that slipped past source 1. +- Versions parsed from prose that happen to look semver-ish but aren't releases. + +```bash +gh api repos/NVIDIA/NemoClaw/tags --paginate --jq '.[].name' > /tmp/nemoclaw-tags.txt + +# For each candidate version V: +grep -Fxq "$V" /tmp/nemoclaw-tags.txt || drop_version "$V" +``` + +After validation, **pick the smallest surviving version** as the reported version (most conservative — it maximizes versions-behind). This handles "this bug was first reported on v0.0.6 and still happens on v0.0.10" cleanly: we verify against latest, and if the bug is gone, both reports are addressed. + +If no version survives, drop the issue from the candidate set — we cannot establish "previous version". + +**Variable format for downstream steps.** Set `REPORTED_VERSION` to the **full tag string** (e.g., `REPORTED_VERSION="v0.0.32"`), not just the patch number. Step 8a's installer expects the full tag via the `NEMOCLAW_INSTALL_TAG` env var. + +**Batch cap enforcement.** In batch mode, after Step 3 label filters and the Step 4 version+candidate-rule filters narrow the pool, sort surviving candidates by `(-versions_behind, -age_days)` so the most stale come first, then **slice to the top 15**: + +```bash +# Each candidate has at minimum: number, reported, behind, age_days +SLICED=$(printf '%s' "$CANDIDATES_JSON" | jq ' + sort_by([-(.behind // 0), -(.age_days // 0)]) + | .[0:15]') +SLICED_COUNT=$(printf '%s' "$SLICED" | jq 'length') +TOTAL=$(printf '%s' "$CANDIDATES_JSON" | jq 'length') +echo "Batch run: processing $SLICED_COUNT of $TOTAL eligible candidates (cap: 15)." +[ "$TOTAL" -gt 15 ] && echo " Spillover: $((TOTAL - 15)) candidates deferred to next run; the marker-comment TTL (Step 3) keeps them eligible." +``` + +The slice is the only enforcement of the cap — without it, "Cap at 15" is policy that nothing actually applies. Single-issue mode bypasses the cap entirely (the user explicitly named one issue). + +**NVBugs cross-reference.** Many NV QA bugs include an NVBugs ticket footer like `[NVB#6100043]`. Extract it at the same time as the version so Step 8.5's comment template (and any other comment template that wants to mention it) can include the cross-reference: + +```bash +NVBUGS_REF=$(printf '%s' "$BODY" | grep -oE '\[NVB#[0-9]+\]' | head -1) +``` + +Templates ignore this when empty. When present, the comment must note that closing the GitHub issue does not propagate to NVBugs and QA needs to update the ticket separately. + +### Implementer note: regex-pipeline pitfalls + +Three real failure modes surfaced during the v1 dry-run. Test each before trusting your implementation: + +1. **Empty-match handling.** A naive pipeline like `[scan(regex)] | first | .[0] | tonumber // fallback` silently dropped 9 real candidates (e.g. #2861 with `NemoClaw 0.0.32`, #2604 with `NemoClaw: 0.0.28`). When `scan` returns no matches, `[]` flows in, `first` returns null, `null | .[0]` errors, and `//` does not propagate cleanly through the error. Bind each pass to a named variable, coalesce at the end: + + ```text + primary := first nemoclaw-anchored match in body (or null) + result := primary ?? null + ``` + + Then explicitly test against a body with **no** version mention. + +2. **Capture-group consistency.** A regex without a capture group (e.g. `\bv\d+\.\d+\.\d+\b`) makes `scan` emit raw strings; with a capture group (e.g. `\b(v\d+\.\d+\.\d+)\b`), `scan` emits arrays. Mixing the two within one pipeline (`first | .[0]?`) works for one and silently fails for the other. Use capture groups consistently across all branches. + +3. **Variable scoping in `select(...)`.** A line like `select($tags | index(.))` rebinds `.` to `$tags` inside the parens, so `.` no longer refers to the surrounding label being checked. Bind first: `. as $lbl | select($tags | any(. == $lbl))`. Symptom in this dry-run: the future-release label `v0.0.35` passed validation that should have rejected it. + +--- diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/environment-and-reproducer.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/environment-and-reproducer.md new file mode 100644 index 00000000000..870e13a504a --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/environment-and-reproducer.md @@ -0,0 +1,260 @@ +<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> +<!-- SPDX-License-Identifier: Apache-2.0 --> + +# verify-stale — Environment and Reproducer Reference + +Use after a candidate passes selection. Classify the environment, prompt for provider credentials when required, extract the reproducer, verify preconditions, and try the local-first path. + +## Contents + +- [Step 5: Classify the Verification Environment](#step-5-classify-the-verification-environment) +- [Step 6: Extract the Reproducer](#step-6-extract-the-reproducer) +- [Step 6.5: Verify Preconditions](#step-65-verify-preconditions) +- [Step 6.7: Try Local Reproduction First](#step-67-try-local-reproduction-first) + +--- + +## Step 5: Classify the Verification Environment + +**CPU vs GPU:** GPU if any of these signals are present, else CPU. + +- Labels: `Platform: GB10`, `Platform: DGX Spark`. +- Body keywords (whole-word, case-insensitive): `nvidia-smi`, `cuda`, `H100`, `A100`, `L40S`, `L4`, `T4`, `GB10`, `DGX`, `vllm`, `tensorrt`. Match as whole words — `inference` and `model serving` are too noisy (e.g. `models.providers.inference.baseUrl` is a config path on CPU bugs, not a GPU need) and intentionally excluded. + +CPU default keeps cost low. Only escalate to GPU when the reproducer needs one. + +**Bug class classification.** In addition to CPU/GPU, classify the bug's verification shape so Step 8 routes to the right rubric. Classes are mutually exclusive — pick the first that matches: + +| Class | Detection heuristic | Routes to | +|---|---|---| +| `performance` | Body or title mentions latency thresholds (`P50`, `P90`, `ms`, `seconds`, `slow`, `hangs`, `timeout` with a numeric value), or mentions `memory leak` / `over time` / `eventually` | Step 8e (multi-run distribution rubric) | +| `rebuild-cycle` | Body mentions `rebuild`, `recreate`, `restart`, `pod recreate`, `across rebuilds`, `after restart`, `survives a destroy` | Step 8f (run-rebuild-rerun harness) | +| `log-only` | Body's symptom is logs-not-stdout: `see lots of error in <X> log`, `os.networkInterfaces guard errors`, anything pointing at a specific log file rather than the reproducer's stdout/stderr | Step 8b's match rubric extended with log-scraping | +| `functional` (default) | Everything else — exit code + stdout/stderr matching | Step 8b standard rubric | + +Most bugs are `functional`. The other three classes need verification harnesses that the standard rubric can't produce honestly — e.g., one clean run of a perf reproducer doesn't tell you the p50 budget was met; one onboard run doesn't tell you a config survives a rebuild. Set `BUG_CLASS=<class>` so downstream steps can branch. + +**Provider classification.** Some bugs are tied to a specific inference provider (NVIDIA NIM, Gemini, Anthropic, OpenAI) and won't reproduce faithfully under Ollama substitution. Classify which provider the issue references so downstream steps either prompt for the right API key or accept the substitution penalty: + +| Detection signal | Provider | +|---|---| +| `Provider: NVIDIA` label, body mentions `NVIDIA NIM`, `build.nvidia.com`, `nvapi-...`, `NVIDIA_API_KEY`, or `NEMOCLAW_PROVIDER=build` | `nim` | +| `Provider: Gemini` label, body mentions `Gemini`, `gemini-flash`, `gemini-pro`, `GEMINI_API_KEY` | `gemini` | +| `Provider: Anthropic` / `Provider: AWS` (Bedrock) labels or matching keywords | `anthropic`/`bedrock` | +| `Provider: Ollama`, body mentions `ollama` or `NEMOCLAW_PROVIDER=ollama`, or no provider mentioned at all | `ollama` (default) | + +Set `BUG_PROVIDER=<provider>`. + +**Required-API-key prompt.** When `BUG_PROVIDER` is anything other than `ollama` AND the bug's reproducer actually exercises inference (not pure CLI surface or sandbox build), the skill MUST stop here and prompt the maintainer interactively before any Brev cost is incurred: + +```text +The reporter's reproducer uses the <provider> provider, which requires a real API key +to verify faithfully. Three options: + + 1. Provide an API key via file (NEVER on the command line — keys in argv are + visible in `ps -ef` to anyone with shell access on either machine). Write + the key to a 600-perm file on your laptop: + + printf '%s' '<your-key>' > ~/.nvidia-api-key + chmod 600 ~/.nvidia-api-key + + The skill copies the file to the Brev box via `brev copy` (encrypted SSH), + reads it inside the box with `NVIDIA_API_KEY=$(cat ~/.nvidia-api-key)`, + and never puts the value on a command line. Box deletion removes the file + from the box; you should `rm ~/.nvidia-api-key` on your laptop after the + run. + + 2. Substitute Ollama and accept the -30 confidence penalty (per Step 8a.5). The + verdict will be capped because we're not exercising the real provider's code + path. + + 3. Skip this issue. Mark `verify-inconclusive` with the reason "requires <provider> + API key — not provided in this run." + +Choose 1, 2, or 3: +``` + +This prompt blocks before Step 7 provisions a box. Don't burn cost on a verification path the maintainer hasn't agreed to. + +**API-key propagation pattern (for option 1).** Argv exposure is a two-layer problem and the file-based pattern must extend to both layers. + +**Layer 1 — local → Brev (surfaced #2604).** Passing the key as `NVIDIA_API_KEY=<value> brev exec ...` puts the literal value in the brev exec process's argv on the maintainer's laptop *and* on the Brev box (since brev exec serializes argv to the remote shell). Visible in `ps -ef` on both ends for the duration of the run. Use file-based copy: + +```bash +# After Step 6.5 preconditions, copy the local key file to the Brev box. +[ -f ~/.nvidia-api-key ] && brev copy ~/.nvidia-api-key "$INSTANCE_NAME":~/.nvidia-api-key +brev exec "$INSTANCE_NAME" "chmod 600 ~/.nvidia-api-key 2>/dev/null || true" +``` + +**Layer 2 — on-box subshell (surfaced #2611).** Inside scripts running on the Brev box, the outer shell reads the key from `~/.nvidia-api-key` cleanly, but a *naive* inner subshell call leaks it back into argv: + +```bash +# WRONG — the double-quoted outer heredoc interpolates $NVIDIA_API_KEY at +# script-eval time, so the literal nvapi- value lands in `sg docker -c "..."`'s +# argv and shows up in `ps -ef` on the box for the whole onboard window. +NVIDIA_API_KEY=$(cat ~/.nvidia-api-key) +sg docker -c " + export NVIDIA_API_KEY='$NVIDIA_API_KEY' # ← argv leak + nemoclaw onboard ... +" + +# RIGHT — escape the $ so the outer shell does not interpolate, and let the +# inner subshell read the file itself. Argv contains the command string +# `cat ~/.nvidia-api-key`, not the value. +sg docker -c " + export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key) + nemoclaw onboard ... +" +``` + +The same rule applies to any `bash -c "..."`, `bash -lc "..."`, `su -c "..."`, `ssh host "..."`, or other invocation that takes a command string as a single argv element: **never interpolate the key into the string at the outer shell's eval time**. Read the file inside the inner shell so the value lives in env-vars, never in argv. + +Cleanup: when the trap fires `brev delete`, the box (and the key file on it) goes away. On the maintainer's laptop, the file persists until they `rm ~/.nvidia-api-key` — Step 12's session log should remind them. **If the key was previously propagated via cmdline (pre-fix at either layer), treat it as exposed and rotate.** + +**Pure-CLI / pure-sandbox-build bugs are exempt** — those don't actually exercise inference, so the provider doesn't matter even if the issue body mentions one. Heuristic: if Step 6.7's local-first predicate would have fired (no sandbox state, no model server interaction), skip the prompt. + +--- + +## Step 6: Extract the Reproducer + +Extract whatever's available from the issue body. The decision about *whether the reproducer is good enough* lives in Step 8 (validate-on-baseline), not here. + +NV QA files most bugs through an HTML form, so issue bodies are typically a mix of `<pre>...</pre>` blocks and tables — not markdown fenced code blocks. Extraction must handle both shapes. + +1. **Verbatim:** the first markdown fence (```` ``` ```` or ```` ~~~ ````) **or** HTML `<pre>` block containing a `nemoclaw` invocation. Strip surrounding tags and unescape HTML entities before saving to `./reproducer.sh`. No confidence penalty (yet). +2. **No verbatim block found:** leave `./reproducer.sh` absent. Step 8b will synthesize from the issue body on demand and apply the **−30 synth penalty** at that point. + +A robust extractor handles both shapes with the body fetched as JSON. The "anchor word" — what marks a block as a reproducer — must include `nemoclaw`, `openclaw`, AND `openshell`. Issue #2592 surfaced this gap: its reproducer was `openclaw channels add telegram` run inside the sandbox; a `nemoclaw`-only regex would have missed the verbatim block and forced the run through Step 8c synth-repro with a -30 penalty: + +```bash +BODY=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json body -q .body) + +REPRODUCER=$(printf '%s' "$BODY" | python3 -c ' +import re, sys, html +b = sys.stdin.read() +# Anchor word: any of nemoclaw / openclaw / openshell. Issue bodies use whichever +# tool the reporter ran (host-side nemoclaw vs in-sandbox openclaw vs openshell CLI). +ANCHOR = r"(?:nemoclaw|openclaw|openshell)" +m = re.search(rf"```(?:bash|sh)?\n(.*?{ANCHOR}.*?)\n```", b, re.S) +if not m: m = re.search(rf"~~~(?:bash|sh)?\n(.*?{ANCHOR}.*?)\n~~~", b, re.S) +if not m: m = re.search(rf"<pre[^>]*>(.*?{ANCHOR}.*?)</pre>", b, re.S) +if m: + text = re.sub(r"<[^>]+>", "", m.group(1)) + print(html.unescape(text).strip()) +') + +[ -n "$REPRODUCER" ] && printf '%s\n' "$REPRODUCER" > ./reproducer.sh +``` + +The "give up immediately" path is gone. Synthesis happens at validation time so it has the baseline transcript to react to, not just the issue body in isolation. The give-up decision now lands in Step 8c when synth fails to produce a script that actually exposes the bug. + +--- + +## Step 6.5: Verify Preconditions + +Confirm CLI dependencies are available, `brev` is authenticated, and the install URL resolves before paying any cost. Credentials live in `~/.brev/credentials.json` and are reused across shells under the same OS user, so once authenticated the auth check is a no-op until the token expires. + +```bash +# CLI deps — fail fast if anything later in the skill needs them but they're missing. +for cmd in gh brev jq python3 curl; do + command -v "$cmd" >/dev/null 2>&1 || { echo "ERROR: missing required dependency: $cmd"; exit 1; } +done + +# gh identity — every comment posted by Step 10 lands under whatever account `gh` is currently +# authenticated as. Surface that explicitly so the maintainer notices before a public comment +# lands under the wrong handle (this matters when `gh` is multi-token, after a recent re-auth, +# or when running under a service-account hostname). +GH_IDENTITY=$(gh api user --jq .login 2>/dev/null) +if [ -z "$GH_IDENTITY" ]; then + echo "ERROR: gh CLI is not authenticated. Run: gh auth login # then re-run this skill" + exit 1 +fi +echo "gh identity: @$GH_IDENTITY — comments posted by this run will appear under this handle" + +# gh 'project' scope — Step 10 moves fixed-on-latest issues to "Needs Review" on Project 199. Warn if missing. +gh auth status 2>&1 | grep -q "'project'" || echo "[verify-stale] WARN gh missing 'project' scope — Step 10 tracker move will skip. Fix: run 'gh auth refresh -h github.com -s project' in a real terminal." + +# Brev auth — short-circuit only after the auth check, not before. +# When auth fails, give the user a directive recipe (the browser-flow path is +# what works from non-TTY harnesses like Claude Code, not the headless options). +brev ls --json >/dev/null 2>&1 || { + cat <<'MSG' + +ERROR: Brev not authenticated. ~/.brev/credentials.json is missing or the token expired. + +What to do (works from any harness, including non-TTY agent contexts): + + 1. Open a separate Terminal on your laptop. + 2. Run: brev login + A browser opens; complete the auth flow; the CLI exits on success. + 3. Come back here and re-run this skill. Credentials persist to + ~/.brev/credentials.json and every subsequent `brev` call picks them up. + +Headless / no-browser alternatives (when option 1 isn't available): + - brev login --skip-browser # prints a URL, paste into any browser + - brev login --token "$BREV_API_TOKEN" # non-interactive; same env var used + # by test/e2e/brev-e2e.test.ts + +MSG + exit 1 +} + +# Repo labels exist — Step 8.5 / Step 10 can't apply a label that doesn't exist. Check +# canonical label names against the live repo so a mismatch fails fast (issue #2168 hit this: +# spec called the label `wontfix`, but the actual repo label is `status: wont-fix`). +EXPECTED_LABELS=("fixed-on-latest" "verify-inconclusive" "status: wont-fix") +LIVE_LABELS=$(gh label list --repo NVIDIA/NemoClaw --limit 200 --json name --jq '.[].name') +for label in "${EXPECTED_LABELS[@]}"; do + printf '%s\n' "$LIVE_LABELS" | grep -Fxq "$label" || { + echo "ERROR: expected label not on repo: '$label'" + echo " create it with: gh label create '$label' --repo NVIDIA/NemoClaw" + exit 1 + } +done + +# Install URL reachable — fails fast instead of mid-Brev-run if the host is down or the URL changed. +# The default is the public Akamai-hosted entry (301-redirects to the actual installer). The +# `nemoclaw.nvidia.com` host that earlier drafts pointed to is NVIDIA-internal and does not +# resolve from Brev; surfaced during the #2007 e2e run. +INSTALL_URL=${NEMOCLAW_INSTALL_URL:-https://www.nvidia.com/nemoclaw.sh} +curl -fsI "$INSTALL_URL" >/dev/null 2>&1 || { + echo "ERROR: install URL not reachable: $INSTALL_URL" + echo " - Check https://www.nvidia.com/nemoclaw.sh is up (the default Akamai-hosted entry)." + echo " - Override with NEMOCLAW_INSTALL_URL=<alternate-url> if your team mirrors the installer." + echo " - Then re-run this skill." + exit 1 +} +``` + +--- + +## Step 6.7: Try Local Reproduction First + +For pure-CLI reproducers (no sandbox state, no GPU, no integration tokens), try locally before paying for a Brev box. The evidence is identical — `nemoclaw <args>` on a maintainer laptop produces the same exit code and stdout as on a fresh Brev VM, modulo platform differences — and the run is free. + +**Predicate** — local-first applies if **all** of these hold: + +- Reproducer is a sequence of `nemoclaw <args>` invocations only. No `docker`, `kubectl`, `curl`, `npm`, networking setup, or filesystem fixtures. +- Issue has no `Sandbox`-only or `Docker` label and no GPU signal from Step 5. +- `which nemoclaw` resolves on the maintainer's machine and `nemoclaw --version` reports a build at or past `$LATEST` (a build between `$LATEST` and `$LATEST+main` is fine — these only differ by unmerged WIP). +- Maintainer is on Linux or macOS. Windows local repros are out of scope (per Step 3 platform skip rules). + +**If the predicate fires:** + +```bash +LOCAL_VERSION=$(nemoclaw --version 2>&1) +LOCAL_TRANSCRIPT=$(mktemp) +{ time bash reproducer.sh; } >"$LOCAL_TRANSCRIPT" 2>&1 +LOCAL_EXIT=$? +echo "Local: $LOCAL_VERSION, exit $LOCAL_EXIT" +``` + +Compare local result to the issue's "Actual Result" section using the same match rubric Step 8b applies on baseline: + +- **Local matches the issue symptom exactly** (same exit code + same diagnostic output) AND the symptom is the post-fix expected output → skip Brev. Use the local transcript as the verified-on-latest evidence. Step 10's comment must say `Environment: local install (<version>) — Brev provisioning skipped, outcome deterministic from CLI surface alone`. +- **Local result differs from the reported "Actual Result"** → continue to Step 7 and run on Brev. The local environment may be a confound (different OS, dirty config, partial build); remote confirms. +- **Local repro errors out for environmental reasons** (`nemoclaw: command not found`, npm link broken) → continue to Step 7. Treat as inconclusive locally, not a verification failure. + +**If the predicate does not fire:** proceed to Step 7 normally. Most sandbox-touching bugs need Brev. + +--- diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md deleted file mode 100644 index 5291c668a63..00000000000 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/execution-and-comment.md +++ /dev/null @@ -1,1173 +0,0 @@ -<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> -<!-- SPDX-License-Identifier: Apache-2.0 --> - -# verify-stale — Execution, Scoring, and Comment Reference - -This file holds Steps 7–12 of the `nemoclaw-maintainer-verify-stale` workflow (everything after the candidate has cleared the local-first short-circuit in Step 6.7). The parent SKILL.md handles candidate filtering, version parsing, environment classification, reproducer extraction, preconditions, and the local-first decision; once a Brev run is committed to, follow this file. - -## Contents - -- **[Step 7: Reuse or Provision a Brev Box](#step-7-reuse-or-provision-a-brev-box)** — concurrency cap, runtime CPU SKU picking, file-based API key copy, cleanup trap. -- **[Step 8: Validate on Baseline, Verify on Latest](#step-8-validate-on-baseline-verify-on-latest)** — comprehensive reset, baseline install (Step 8a), reproducer dependency bootstrap (8a.5), brev-exec environment quirks (8a.5b), baseline run (8b), synth-repro retry (8c), latest install (8d), architectural-drift check (8d.5), performance-bug verification (8e), rebuild-cycle verification (8f). -- **[Step 8.5: Detect "Behavior Changed by Design"](#step-85-detect-behavior-changed-by-design)** — three signals, related-failure-mode pre-check, test-coverage check, self-verification, by-design comment template. -- **[Step 9: Score Confidence](#step-9-score-confidence)** — +50 / +25 / +25 / −30 / −50 rubric, baseline-validation cap-at-84, path extraction (commits-touched), PR-search (PR-mention). -- **[Step 10: Compose and Post the Comment](#step-10-compose-and-post-the-comment)** — redaction table (HTML→text pre-pass; JWT/PAT/NVAPI/base64/internal-host/email patterns), comment-authoring principle (300-word ceiling), per-verdict length defaults, mandatory caveats (cap, hardware substitution, verification mode, link self-verify), three templates (fixed/inconclusive, still-reproduces, by-design), unanswered-question prefix and dual @-mention variant. -- **[Step 11: Infra Failure Handling](#step-11-infra-failure-handling)** — sandbox-build rot is the dominant failure mode for any version >5–7 patches behind; cap-at-84 with reporter @-mention is by design. -- **[Step 12: Log to Activity](#step-12-log-to-activity)** — per-issue and per-session entries to `~/development/daily-rhythm/activity/nemoclaw-verify-stale-log.md`. -- **[Cadence](#cadence)** — weekly cron + manual single-issue invocation. -- **[Out of Scope (v1)](#out-of-scope-v1)** — auto-close, macOS verification, integration-credential bugs, service-account bot, versioned labels. -- **[Companion Behavior](#companion-behavior)** — `nemoclaw-maintainer-cut-release-tag` sweeps verification labels at release time. - ---- - -## Step 7: Reuse or Provision a Brev Box - -The skill prefers reuse over provisioning. A pool of `verify-stale-*` boxes (CPU and GPU) can be kept warm; reuse the matching one if available, otherwise provision. - -```bash -# Auth + install URL already verified by Step 6.5 — no need to re-check or auto-login here. - -# Determine class from Step 5: "cpu" or "gpu" -INSTANCE_CLASS="cpu" # or "gpu" - -INSTANCES=$(brev ls --json) - -# Look for an existing running verify-stale-* box matching the required class. -# CPU boxes have no .gpu field set; GPU boxes do. -EXISTING=$(echo "$INSTANCES" | jq -r --arg class "$INSTANCE_CLASS" ' - .[]? - | select(.name | startswith("verify-stale-")) - | select(.status == "RUNNING") - | select(($class == "gpu" and (.gpu // "" != "")) - or ($class == "cpu" and (.gpu // "" == ""))) - | .name' | head -1) - -PROVISIONED_NEW=0 - -if [ -n "$EXISTING" ]; then - INSTANCE_NAME="$EXISTING" - echo "Reusing existing verification box: $INSTANCE_NAME" -else - # Concurrency cap: refuse if 4+ verify-stale-* boxes are already running. - # Filter on .status to match the reuse query above — counting non-running boxes - # would falsely block provisioning when prior boxes are stopped but not deleted. - RUNNING=$(echo "$INSTANCES" | jq '[.[]? | select(.name | startswith("verify-stale-")) | select(.status == "RUNNING")] | length') - if [ "$RUNNING" -ge 4 ]; then - echo "ERROR: 4 verify-stale boxes already running. Wait for one to finish or reuse." - exit 1 - fi - - INSTANCE_NAME="verify-stale-${ISSUE_NUMBER}-$(date +%s)" - - if [ "$INSTANCE_CLASS" = "gpu" ]; then - # brev create auto-selects the cheapest GPU meeting the defaults - # (>=20GB VRAM, >=500GB disk, compute >=8.0). Override with --type if needed. - brev create "$INSTANCE_NAME" - else - # CPU case: pick the cheapest stoppable Linux SKU at runtime so the skill doesn't rot when - # SKUs change. Bias the floor by reproducer-implied memory needs — the cheapest 2 GB SKU - # cannot load a 4.8 GiB Ollama probe, and onboard fails at provider validation before any - # sandbox-creation code runs. Surfaced during the #2007 e2e run (wasted ~25 min on a 2 GB - # box that couldn't load `nemotron-3-nano:4b`). - # - # Memory floor heuristic: - # - Reproducer references Ollama or vLLM or names a model tag (e.g. `nemotron-3-nano:4b`, - # `llama3:8b`) -> floor 16 GB (covers ~5 GB model + sandbox + gateway overhead). - # - Reproducer touches sandbox onboarding without a local model server -> floor 8 GB. - # - Pure CLI-surface bug (no sandbox, no model) -> floor 4 GB. - # Override the auto-pick by exporting VERIFY_STALE_CPU_TYPE if the team has hard preferences. - CPU_RAM_FLOOR=${CPU_RAM_FLOOR:-8} - CPU_TYPE=${VERIFY_STALE_CPU_TYPE:-$(brev search cpu --sort price --json \ - | jq -r --argjson floor "$CPU_RAM_FLOOR" \ - '[.[] | select(.stoppable == true and .ram_gb >= $floor)] | .[0].type')} - [ -n "$CPU_TYPE" ] || { echo "ERROR: no stoppable CPU SKU with >= ${CPU_RAM_FLOOR} GB RAM"; exit 1; } - brev create "$INSTANCE_NAME" --type "$CPU_TYPE" - fi - - PROVISIONED_NEW=1 -fi - -# Cleanup runs on success, error, and SIGINT. -# Delete only what we provisioned. Reused boxes stay warm for next time. -# `brev delete` is non-interactive by default — there is no --yes flag, and passing one errors. -echo ">>> Brev instance: $INSTANCE_NAME (provisioned_new=$PROVISIONED_NEW; manual cleanup: brev delete $INSTANCE_NAME)" -trap '[ "$PROVISIONED_NEW" = "1" ] && brev delete "$INSTANCE_NAME" >/dev/null 2>&1 || true' EXIT -``` - -Wallclock cap per verification: **60 minutes** default. The cap accommodates two full install passes (baseline + latest), comprehensive resets between them, and any reproducer dependency bootstrapping (Step 8a.5) — most of which run sequentially against a single Brev box. Bugs that genuinely require more than an hour to manifest fall out of v1 scope; if a provisioned box isn't ready in time, abort and treat as an infra failure (Step 11). - -The previous design had a 25-min default with a 60-min extension for time-sensitive bugs (`memory leak`, `over time`, etc.). That split optimised for the wrong constraint — most issues fit comfortably under 60 min, and the keyword-based extension forced re-runs whenever a real install or bootstrap took longer than the optimistic 25-min budget. Single 60-min cap removes that paper cut. - ---- - -## Step 8: Validate on Baseline, Verify on Latest - -Two-pass design. - -- **Baseline pass (8a–8c):** install the **reported version**, run the reproducer, confirm it actually exposes the bug as described. This is the gate that proves the script is real. -- **Latest pass (8d):** install **latest**, run the validated reproducer. This is what the confidence score is built on. - -Without the baseline gate, a clean run on latest is ambiguous: maybe the bug really got fixed, maybe the script was never capable of triggering it. The baseline disambiguates. - -### Comprehensive reset (run before each install) - -NemoClaw spawns OpenShell sandboxes (containers), runtime services, and listening processes. A naive `rm -rf ~/.nemoclaw` doesn't clean those — the latest install would inherit baseline state and contaminate the result. Use this fuller reset between installs: - -```bash -RESET=$(cat <<'SCRIPT' -nemoclaw destroy --all --force 2>/dev/null || true -# Anchor pkill patterns to "/nemoclaw" / "/openshell" path components so the kill doesn't -# match unrelated processes that happen to mention these strings (including the agent -# harness running this skill if its working dir contains the word). -pkill -9 -f '/nemoclaw([[:space:]]|$)' 2>/dev/null || true -pkill -9 -f '/openshell([[:space:]]|$)' 2>/dev/null || true -docker ps -a --filter "name=openshell-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true -docker ps -a --filter "name=nemoclaw-" -q 2>/dev/null | xargs -r docker rm -f 2>/dev/null || true -# Sandbox state lives in ~/.openclaw (default-writable since #2227); ~/.nemoclaw holds CLI state. -# Wipe both so the latest install starts clean. -rm -rf ~/.nemoclaw ~/.openclaw 2>/dev/null -sudo -n rm -f /usr/local/bin/nemoclaw 2>/dev/null || true -sudo -n rm -rf /usr/local/lib/nemoclaw 2>/dev/null || true -for port in 8080 18789 9119; do fuser -k -n tcp $port 2>/dev/null || true; done -true -SCRIPT -) -``` - -Idempotent — fails silently when there's nothing to clean. Run via `brev exec "$INSTANCE_NAME" "$RESET"` before 8a's install and again before 8d's install. - -**Sudo precondition.** All `sudo` invocations use `sudo -n` (non-interactive) so they fail fast instead of hanging on a password prompt. The skill assumes the Brev image's default user has passwordless sudo configured — Brev's stock images do; custom images may not. If `sudo -n` fails, the binary cleanup is best-effort and a stale `/usr/local/bin/nemoclaw` may persist. The user-local install path (`~/.nemoclaw`) is fully reset regardless. - -### Step 8a: Install reported version - -The installer accepts the target ref via the `NEMOCLAW_INSTALL_TAG` env var (verified against `install.sh` source — defaults to `latest` if unset). It is **not** a `--version` flag. - -```bash -brev exec "$INSTANCE_NAME" "$RESET" - -# Pass the provider env vars through so install.sh's bundled `[3/3] Onboarding` step -# doesn't fall back to the default `build` (NIM) provider — which requires NVIDIA_API_KEY -# and otherwise fails the install with a misleading error. When NEMOCLAW_PROVIDER=ollama -# (the common case), the bundled onboard uses the local Ollama we set up in Step 8a.5 -# and either succeeds (ideal) or fails on a real Dockerfile/sandbox-build issue (which -# is what we want to detect). Pass NVIDIA_API_KEY only if the maintainer provided one -# at Step 5's prompt. -# Read NVIDIA_API_KEY from ~/.nvidia-api-key on the BOX (not from this shell's argv). -# The Step 5 propagation block already brev-copy'd the key file with 600 perms. -brev exec "$INSTANCE_NAME" " - if [ -f ~/.nvidia-api-key ]; then export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key); fi - NEMOCLAW_INSTALL_TAG=$REPORTED_VERSION \ - NEMOCLAW_NON_INTERACTIVE=1 \ - NEMOCLAW_PROVIDER=${NEMOCLAW_PROVIDER:-ollama} \ - NEMOCLAW_MODEL=${NEMOCLAW_MODEL:-nemotron-3-nano:4b} \ - NEMOCLAW_SANDBOX_NAME=verify-stale-install \ - bash -c 'curl -fsSL $INSTALL_URL | bash' -" || BASELINE_INSTALL_FAILED=1 - -# Verify the resolved install version matches the requested version. This guards against the -# `VAR=val curl ... | bash` shell-scoping footgun where the env var binds to curl, not the -# downstream bash, and the install silently falls through to "latest". Surfaced during a -# rot-debugging investigation where v0.0.36 was silently installed when v0.0.26 was requested -# and several minutes of "convincing" output ran before anyone noticed. Always print the -# resolved state, never trust the requested state. -RESOLVED=$(brev exec "$INSTANCE_NAME" "bash -lc 'nemoclaw --version'" 2>&1 | tail -1) -echo "[verify-stale] baseline requested: $REPORTED_VERSION; resolved: $RESOLVED" -case "$RESOLVED" in - *"$REPORTED_VERSION"*) ;; # match — proceed - *) - echo "ERROR: baseline install resolved to '$RESOLVED' but $REPORTED_VERSION was requested." - echo " Common cause: env-var scoping in the install command. Verify the env vars are on" - echo " the BASH side of the curl|bash pipe, not the curl side. Setting" - echo " BASELINE_INSTALL_FAILED=1 to prevent verifying against the wrong version." - BASELINE_INSTALL_FAILED=1 - ;; -esac - -# The bundled onboard creates a sandbox name we don't want carrying through to the reproducer. -# Use a hyphen-only name (NemoClaw's name validator rejects underscores). Destroy it so the -# reproducer starts from a clean state. -brev exec "$INSTANCE_NAME" "sg docker -c 'nemoclaw destroy --all --force 2>/dev/null || true'" -``` - -If install fails (old releases rot — installer URLs, deps, OS images all drift over time, or the in-image Dockerfile patch step asserts against a code shape that's since changed), set `BASELINE_INSTALL_FAILED=1` and **skip 8b/8c**, going straight to 8d. Note "baseline-install-skipped" or "baseline-build-skipped" in the final comment depending on which phase rotted. Step 9's scoring rule handles the degraded mode (cap at 84). - -**The reproducer's own `nemoclaw onboard` (Step 8b) must pass `--fresh`.** If install.sh's bundled onboard was in an in-progress or failed state when we destroyed the install sandbox, the reproducer's onboard would error with `Previous onboarding session failed. Re-run with --fresh to discard it`. `--fresh` ensures a clean start. - -### Step 8a.5: Bootstrap reproducer dependencies - -Brev's stock CPU images ship with NemoClaw installable but not the broader ecosystem the reproducer may need — local model servers (Ollama, vLLM), inference providers, third-party CLIs. **Default to maximum faithfulness: install the actual dependency the reporter used rather than substituting a stub.** Substituting trades faithfulness for speed; that trade is rarely worth it on a 60-min budget, and it almost always introduces a confound that makes the verdict less trustworthy. - -**When to bootstrap (not substitute):** - -- The reproducer references a specific model/server runtime (`NEMOCLAW_PROVIDER=ollama`, `NEMOCLAW_PROVIDER=vllm`, etc.). -- The reproducer references a specific model name with a tag (`nemotron-3-nano:4b`, `llama3:8b`, etc.). -- The reporter's environment in the issue body shows a configured provider (e.g., `OpenShell CLI: 0.0.26` plus an Ollama running on host). - -**When to substitute (with -30 penalty):** - -- Provider requires an API key the skill cannot safely supply (NIM, OpenAI, Anthropic, etc.). Stubbing a key won't pass validation faithfully and a real key shouldn't sit in a verify-stale run. Apply the -30 penalty (treat as synth-repro per Step 8b) and document the substitution in the comment. -- The bug is *provably* independent of the dependency (e.g., a CLI argument-parsing bug that errors before any provider runs). Note this explicitly in the comment. - -**Canonical bootstraps:** - -```bash -# Ollama + a specific model. -# The Ollama installer registers a systemd service (`ollama.service`) so the -# daemon survives between brev exec calls. -brev exec "$INSTANCE_NAME" "curl -fsSL https://ollama.com/install.sh | sh" -brev exec "$INSTANCE_NAME" "sudo systemctl start ollama && sleep 3" -brev exec "$INSTANCE_NAME" "ollama pull <model>" -brev exec "$INSTANCE_NAME" "ollama list" # confirm before continuing -``` - -```bash -# vLLM + a model (HuggingFace-hosted). -brev exec "$INSTANCE_NAME" "pip install --quiet vllm" -brev exec "$INSTANCE_NAME" "nohup python -m vllm.entrypoints.openai.api_server --model <model> --host 127.0.0.1 --port 8000 >/var/log/vllm.log 2>&1 &" -brev exec "$INSTANCE_NAME" "sleep 30 && curl -fsS http://127.0.0.1:8000/v1/models" -``` - -Bootstrap **once before Step 8b's baseline run** and reuse for Step 8d's latest run. Don't reset Ollama/vLLM state between baseline and latest in the comprehensive reset — model downloads are expensive and unrelated to the NemoClaw install. Adjust the reset script to skip these external services explicitly if needed. - -**If bootstrap fails** (network issue pulling the model, service won't start, etc.), this is an infra failure — abort to Step 11. Do not silently substitute; the user opted into faithfulness for a reason. - -**Ollama coverage table.** Ollama is the default provider for verification runs because it's free, local, and self-hosted. It covers most bug classes faithfully but not all. Use this table to decide whether Ollama is sufficient or whether Step 5's API-key prompt should fire: - -| Bug class | Ollama covers? | Notes | -|---|---|---| -| CLI surface (subcommand parsing, flag handling, oclif dispatch) | ✓ Always | Provider not exercised | -| Sandbox structure (build, file permissions, mounts, layout) | ✓ Always | Provider not exercised | -| Networking / policy (port forwards, NAT, egress rules, channels guards) | ✓ Always | Provider not exercised | -| Generic inference flow (does an agent turn complete, does the proxy route correctly) | ✓ Usually | Ollama can fail in the same shape as NIM/Gemini for most flow bugs | -| Provider-specific behavior (`Provider: NVIDIA` symptom, NIM-only error handling, `Provider: Gemini` quirks) | ✗ No | Different code paths; substitution doesn't exercise the bug | -| Model-specific behavior (`gemini-flash-3-preview` doesn't handle prompt X, `nemotron-3-nano:4b` works fine) | ✗ No | Wrong model = wrong outputs | -| Ollama-shape-specific (#2519 "Ollama-local 401" — local-vs-networked Ollama config) | △ Sometimes | A generic Ollama install may or may not reproduce; may need specific configuration | -| Performance / latency on specific silicon | ✗ No | Hardware substitution caveat (Step 10) and Step 8e perf rubric apply | -| Quota / rate-limit / API-key validation | ✗ No | Ollama doesn't have those failure modes | - -When the table says ✗ No or △ Sometimes, Step 5's API-key prompt fires. When it says ✓, proceed with Ollama and skip the prompt. - -### Step 8a.5b: Brev exec environment quirks - -Two non-obvious gotchas surfaced during the #2007 e2e run that every subsequent `brev exec` call has to handle. Encode them once here so reproducer scripts don't have to relearn each time. - -**PATH does not include `~/.local/bin` in non-login shells.** `nemoclaw`'s installer drops a shim at `~/.local/bin/nemoclaw` and updates PATH via `~/.bashrc` / `~/.profile`. `brev exec` spawns non-login, non-interactive shells that don't source those files, so a bare `brev exec "$INSTANCE" "nemoclaw --version"` returns `command not found` on a freshly-installed box. Fix: every reproducer script must explicitly export PATH at the top, OR every `brev exec` call must wrap with `bash -lc '...'`. - -```bash -# Reproducer scripts: prepend this line. -export PATH="$HOME/.local/bin:$PATH" - -# Or equivalently when calling brev exec ad-hoc: -brev exec "$INSTANCE" "bash -lc 'nemoclaw --version'" -``` - -**Docker group requires `sg docker -c '...'` after `usermod -aG`.** Adding the user to the `docker` group (`sudo usermod -aG docker ubuntu`) takes effect for new login sessions, but `brev exec` calls in the same Brev session keep the old gid. The reproducer's `nemoclaw onboard` will fail with `permission denied while connecting to /var/run/docker.sock` unless the call runs in a subshell with the docker group active. - -```bash -# Reproducer execution: wrap with sg docker. -brev exec "$INSTANCE" "sg docker -c 'bash ~/reproducer.sh'" -``` - -Both patterns appear in the canonical setup script committed alongside the skill (or are encoded in your reproducer wrapper). Don't rely on the user discovering them mid-run. - -**`openshell sandbox exec` argument-order footgun.** When the reproducer needs to run a command *inside* the sandbox (channels-guard checks, in-sandbox file inspection, etc.), the correct non-interactive form uses `-n <name>` and a `--` separator: - -```bash -# Correct: -openshell sandbox exec -n ai -- bash -c 'source /sandbox/.bashrc; openclaw channels add telegram; echo "EXIT=$?"' - -# Wrong (silently auto-detects sandbox by "last used", stuffs the leftover positional -# `ai` into bash's $0, prints "/bin/bash: line 1: ai: command not found" — the -# reproducer appears to fail but actually never ran inside the sandbox at all): -openshell sandbox exec ai bash -c '...' -``` - -Issue #2592's first run hit this — wasted ~15 min before the maintainer noticed. Always use the `-n <name> -- <cmd>` form when the reproducer touches in-sandbox commands. - -**`brev exec` SSH-drop re-execution guard.** Brev's CLI silently retries from the top when the SSH connection drops mid-run, producing two parallel reproducer executions (we hit this on #2592 — one onboard process clobbered another's state, and both got billed). Use a sentinel file in the reproducer wrapper to make the script idempotent: - -```bash -# At the top of the reproducer wrapper script: -SENTINEL=~/.verify-stale-running -if [ -f "$SENTINEL" ]; then - echo "ERROR: another verify-stale run is in progress (sentinel: $SENTINEL)." - echo " If you're sure no other run is active, rm $SENTINEL and re-invoke." - exit 1 -fi -trap 'rm -f "$SENTINEL"' EXIT -touch "$SENTINEL" -``` - -The sentinel survives an SSH drop because it lives on the Brev box's filesystem; the trap removes it on script exit. A second `brev exec` invocation that tries to retry from the top will hit the sentinel and bail instead of double-running. - ---- - -### Step 8b: Run reproducer on baseline, compare to issue symptom - -If `./reproducer.sh` exists (verbatim from Step 6), run it. Otherwise synth on demand from the issue body (apply −30 penalty now, locked in for the rest of the run). - -**Interactive subcommand handling.** Many `nemoclaw onboard` / `nemoclaw configure` invocations prompt for input and will hang in a non-interactive shell. Auto-detect such subcommands in the script and apply, in order: - -1. Add `--non-interactive` if the version supports it. -2. Add `--dangerously-skip-prompts` (issue #2168 confirmed this exists for at least some Jetson paths). -3. Pre-feed answers via stdin: `printf 'yes\n\n\n' | nemoclaw onboard ...` - -If none work, route the script to Step 8c (synth-repro) so the LLM can rewrite it using non-interactive equivalents. - -```bash -# `brev exec` spawns a non-login shell, so ~/.local/bin (where the nemoclaw binary lives -# after install) is not on PATH unless we export it. The reproducer script itself must -# use `sg docker -c '...'` blocks for any Docker-touching command — Step 8a.5b covers -# that requirement; double-wrapping with sg docker on the outer call breaks nested-quote -# escaping in some bash versions. -brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh -brev exec "$INSTANCE_NAME" 'export PATH="$HOME/.local/bin:$PATH" && bash ~/reproducer.sh' 2>&1 | tee ./baseline-transcript.log -``` - -**Log-scraping (when `BUG_CLASS=log-only`).** Some bugs describe symptoms that show up in internal log files, not the reproducer's stdout/stderr — e.g., #1642 "see lots of error in openclaw log," #2611 "os.networkInterfaces guard errors." After running the reproducer, also pull the relevant logs from inside the sandbox and search them for the issue's symptom phrase: - -```bash -# Common NemoClaw / OpenClaw / OpenShell log paths inside the sandbox. -brev exec "$INSTANCE_NAME" "sg docker -c 'cat ~/.openclaw/logs/*.log /var/log/nemoclaw/*.log 2>/dev/null'" \ - | tee ./baseline-logs.log - -# Search the log capture for the issue's symptom phrase too, not just the transcript. -grep -F "<symptom phrase from issue body>" ./baseline-logs.log -``` - -For functional bugs the reproducer's stdout is sufficient; for log-only bugs the transcript may be clean but the log capture has the symptom. Both halves feed into the match rubric below. - -**Flake-detection retry.** Even for `functional` bugs, race-prone reproducers (TUI rendering, network policy negotiation, concurrent sandbox state) can produce inconsistent results. Run baseline three times if the first run shows the symptom inconsistently — same script, same env, just three back-to-back invocations. If the three runs disagree, that's signal: - -| 3-run baseline result | Verdict | -|---|---| -| All three reproduce the symptom | Strong baseline match → continue to 8d | -| All three are clean (no symptom) | Reproducer doesn't expose the bug on baseline → Step 8c synth-repro | -| Mixed (1 or 2 of 3 show the symptom) | Flake-prone reproducer. Note "flake suspected" in the comment; apply −25 to Step 9 score; downgrade `+50 latest clean` to `+25` because a clean latest run could just be the lucky path of an intermittent bug | - -Skip flake retry for `performance` and `rebuild-cycle` classes — those have their own multi-run rubrics in Steps 8e and 8f. - -**Match rubric.** LLM compares `baseline-transcript.log` to the issue's "Actual result" / error description. Match criteria, in order: - -1. **Exit code agrees** with what the issue describes (non-zero if issue describes a failure, zero if issue describes a wrong-output bug). Necessary but not sufficient. -2. **Symptom phrase match:** transcript contains a key error phrase from the issue (e.g., issue says `Permission denied on generate-openclaw-config.py`, transcript says `EACCES: permission denied, open '...generate-openclaw-config.py'` — semantic equivalence counts). -3. **Distinguish bug from infra noise:** generic network / DNS / auth errors don't count as a match unless the issue itself describes them. A bug about config parsing that fails at "could not resolve nvidia.com" is an infra failure, not a reproduction. - -**Fallback for issues without an explicit "Actual result" section.** Many bug reports describe a *behavioral* problem rather than a runtime error — e.g., "should default to a stable released version" (#1242), "configuration is not persisted across rebuilds" (#3030). These have no comparable error string. In that case: - -1. Use the issue's **full title + description** as the symptom signal. -2. Match if the reproducer's outcome **contradicts the issue's stated expected behavior** (or matches the stated wrong behavior). E.g., issue says "expected: stable release; actual: nightly", reproducer prints `nightly-build-2026.04.x` → that's a match. -3. If neither error string nor expected-behavior contradiction can be identified, route the script to Step 8c (synth-repro) — let the LLM produce a more diagnostic script that emits something testable. - -- **Match** → reproducer validated. Proceed to 8d. -- **No match** (silent pass, wrong error, infra noise, or no testable outcome): script has gaps. Proceed to 8c. - -### Step 8c: Synth-repro and retry on baseline - -LLM rewrites `./reproducer.sh` using the full issue context (description, environment, symptoms) **plus the baseline transcript** so it can react to what actually happened. Apply **−30 confidence penalty** (or keep it if 8b already applied it for the missing-verbatim case). - -```bash -brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh -brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcript-2.log -``` - -- **Match:** validated (with −30 baked in). Proceed to 8d. -- **Still no match:** mark `verify-inconclusive`. Post a comment that includes both reproducer attempts and both baseline transcripts with the message "couldn't establish a working reproducer for this bug on `$REPORTED_VERSION`." **Skip 8d** — there's nothing to verify on latest. - -### Step 8d: Install latest, run validated reproducer - -```bash -brev exec "$INSTANCE_NAME" "$RESET" -brev exec "$INSTANCE_NAME" " - if [ -f ~/.nvidia-api-key ]; then export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key); fi - curl -fsSL $INSTALL_URL | bash -" - -# Same resolved-version check as Step 8a — guard against env-var scoping or default fallthrough -# silently installing the wrong version. The latest install should resolve to $LATEST. -RESOLVED=$(brev exec "$INSTANCE_NAME" "bash -lc 'nemoclaw --version'" 2>&1 | tail -1) -echo "[verify-stale] latest requested: $LATEST; resolved: $RESOLVED" -case "$RESOLVED" in - *"$LATEST"*) ;; # match — proceed - *) echo "WARN: latest install resolved to '$RESOLVED' (expected match for $LATEST). Proceeding but flag in comment." ;; -esac - -# OpenShell version pin — surfaced from #1642's e2e run. Latest's blueprint.yaml may set -# `max_openshell_version` below what the OpenShell installer would otherwise grab. The -# baseline phase (Step 8a) installed whichever OpenShell was current at reported-version, -# which can be newer than latest's cap (e.g., reported v0.0.6 → installed openshell 0.0.37, -# latest v0.0.38 caps at 0.0.36, onboard preflight refuses to run). Re-pin from latest's -# repo so onboard preflight passes; if the new pin is OLDER than the installed binary, -# install-openshell.sh refuses the downgrade — fall back to direct GitHub download. -brev exec "$INSTANCE_NAME" ' - set -e - cd ~/NemoClaw - git fetch --depth 1 origin tag "'"$LATEST"'" 2>&1 | tail -2 - git checkout -- . 2>/dev/null || true - git checkout "'"$LATEST"'" 2>&1 | tail -2 - - MAX_OS=$(grep -E "^max_openshell_version:" nemoclaw-blueprint/blueprint.yaml 2>/dev/null | awk "{print \$2}" | tr -d "\"" | tr -d "v") - CUR_OS=$(openshell --version 2>&1 | grep -oE "[0-9]+\.[0-9]+\.[0-9]+" | head -1 || echo 0.0.0) - echo "[verify-stale] openshell pin: blueprint max=$MAX_OS, currently installed=$CUR_OS" - - if [ -n "$MAX_OS" ] && [ "$(printf "%s\n%s\n" "$CUR_OS" "$MAX_OS" | sort -V | tail -1)" != "$MAX_OS" ]; then - echo "[verify-stale] currently installed openshell ($CUR_OS) is newer than blueprint cap ($MAX_OS) — force-downgrading" - sudo rm -f /usr/local/bin/openshell - cd /tmp - curl -fsSL "https://github.com/NVIDIA/OpenShell/releases/download/v$MAX_OS/openshell-x86_64-unknown-linux-musl.tar.gz" -o openshell-pin.tar.gz - tar -xzf openshell-pin.tar.gz - sudo install -m 755 ./openshell /usr/local/bin/openshell - openshell --version - else - sudo bash scripts/install-openshell.sh 2>&1 | tail -3 - fi -' - -brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh -# Same PATH safeguard as the baseline call — non-login shells don't pick up ~/.local/bin -# automatically. The reproducer's internal `sg docker -c '...'` blocks cover Docker access. -brev exec "$INSTANCE_NAME" 'export PATH="$HOME/.local/bin:$PATH" && bash ~/reproducer.sh' 2>&1 | tee ./latest-transcript.log -``` - -If the install of **latest** fails (e.g. installer regression — see #3058 for a current example), this is an infra failure — see Step 11. Do not score or label the issue. - -If install succeeds, `latest-transcript.log` is the input to Step 9 scoring. - -For interactive debugging when something looks off: - -```bash -brev shell "$INSTANCE_NAME" -``` - ---- - -## Step 8d.5: Architectural-Drift Check - -Cross-version verification compares two moving targets: the reproducer assumes `$REPORTED_VERSION`'s tooling surface, and `$LATEST` may have rewritten the surface entirely. If the *tool* the reproducer relies on (CLI subcommand, output table, log file location) was reworked between the two tags, an "empty / clean output on latest" can mean either "bug fixed" OR "we're looking at a deprecated tracking surface." Without this check, the latter silently registers as the former — a class of false positive. - -**Detection** — pickaxe the diff between tags for the reproducer's tool name and watch for the CLI itself being touched, not just its consumers: - -```bash -# Extract the primary verification command from the reproducer (e.g. "openshell forward list"). -TOOL=$(grep -oE '\b(openshell|nemoclaw)[[:space:]]+[a-z-]+' reproducer.sh | sort -u) - -# Pickaxe each tool name across the version range. -for t in $TOOL; do - echo "=== drift check: $t ===" - git log "$REPORTED_VERSION".."$LATEST" -S"$t" --oneline -- src/ bin/ nemoclaw/src/ 2>&1 | head -5 -done -``` - -If a tool is touched, drift is suspected. - -**Multi-axis verification** — when drift is suspected, do not rely on the reproducer's expected output alone. Pick OS-level surfaces that would show the buggy state regardless of which CLI tracks it. For port-forwarding bugs (the #2007 case), the canonical five-axis pattern: - -| # | Surface | Command | -|---|---|---| -| 1 | Reproducer's stated check | as written in the issue body | -| 2 | Host TCP listeners | `sudo ss -tlnp` | -| 3 | iptables NAT redirects | `sudo iptables -t nat -L -n` | -| 4 | Docker port mappings | `docker ps --format '{{.Names}} {{.Ports}}'` | -| 5 | Active SSH tunnels | `ps -ef \| grep 'ssh.*-L'` | - -Adapt the axes to the bug class. For filesystem bugs: `find`, `lsattr`, `stat`. For network policy bugs: `iptables -L`, container netns, gateway logs. The principle is the same — pick at least three independent surfaces that would each independently show the buggy state if it were present. - -**Action when drift is suspected:** - -- Run the multi-axis pattern after Step 8d's reproducer. -- The verdict requires **every relevant axis to be clean** — not just the reproducer's surface — before claiming `fixed-on-latest`. -- Quote the multi-axis evidence in the Step 10 comment as a table; this is exactly what makes "fixed" defensible when the original tooling no longer reflects the underlying behavior. -- If any axis still shows the buggy state, the bug is NOT fixed even if the reproducer's surface is clean. Escalate to "still reproduces" (Step 9 special case). - -**When drift is NOT suspected** (the reproducer's tool is unchanged in the version range): the reproducer's expected output is sufficient, no multi-axis verification needed. - ---- - -## Step 8e: Performance-Bug Verification (when `BUG_CLASS=performance`) - -Performance bugs (#2598 "10s P50", #2600 "hangs ~2 min", #2733 Ollama tool-call leak over time) can't be answered by the standard exit-code + symptom-phrase rubric — one clean reproducer run doesn't tell you the p50 budget is met; one slow run doesn't tell you the bug still reproduces. Replace Step 8b's match with a measurement-and-distribution rubric: - -1. **Parse the SLA from the issue body.** Extract numeric latency thresholds: `10s P50`, `200ms`, `under 5 seconds`, `~2 min`. Save as `SLA_P50_MS`, `SLA_P90_MS`, etc. If no numeric SLA is in the body, route to Step 8c synth-repro to ask the reporter (via comment) for one — without a target, the verdict is undefined. -2. **Run the reproducer N=10 times** on each side (baseline + latest), capturing per-run latency: - - ```bash - for i in $(seq 1 10); do - /usr/bin/time -f '%e' bash ~/reproducer.sh >/dev/null 2>>./latest-perf.log - done - ``` - -3. **Compute p50 and p90** for both sides. `sort -n ./latest-perf.log | awk 'NR==5'` for p50 of 10 runs. -4. **Match rubric:** - - Latest's p50 within the SLA AND baseline's p50 outside the SLA → bug fixed; same Step 9 scoring (subject to baseline-validation gate). - - Latest's p50 outside the SLA → bug still reproduces (Step 9 special case). - - Latest p50 within SLA AND baseline p50 also within SLA → reproducer doesn't actually exercise the bug; route to Step 8c synth-repro. - -**Hardware-substitution caveat.** Performance numbers are silicon-dependent. When the issue is `Platform: DGX Spark` or `Platform: GB10` and we're measuring on a Brev x86 GPU SKU, the comment must say so explicitly: a Brev p50 of 1.5s on a `H100` does not prove the DGX Spark p50 is fixed. Cap the score at 60 unless the bug is clearly silicon-independent (e.g. an algorithmic regression in user-space JS that would manifest the same on any silicon). - ---- - -## Step 8f: Rebuild-Cycle Verification (when `BUG_CLASS=rebuild-cycle`) - -Rebuild-cycle bugs (#2701 "Pod recreate wipes `/tmp/nemoclaw-proxy-env.sh`," issues describing "configuration is not persisted across rebuilds") only manifest when sandbox state crosses a destroy/recreate boundary. A single onboard run can't trigger the symptom. Replace Step 8b's match with a run-rebuild-rerun harness: - -1. **First onboard.** Run the reproducer once to establish initial state. Capture relevant artifacts (config files, env vars, sandbox metadata) — the issue body usually names what should persist: - - ```bash - brev exec "$INSTANCE_NAME" "sg docker -c 'cat <files-mentioned-in-issue> 2>&1'" | tee ./pre-rebuild.log - ``` - -2. **Trigger the rebuild.** Use `nemoclaw destroy --all --force` followed by `nemoclaw onboard` with the same env vars. Do NOT comprehensive-reset between (the point is to test the destroy/recreate, not start from scratch). - -3. **Re-capture the same artifacts** post-rebuild: - - ```bash - brev exec "$INSTANCE_NAME" "sg docker -c 'cat <same-files> 2>&1'" | tee ./post-rebuild.log - ``` - -4. **Diff and match.** The bug is "X gets wiped / changes / regresses across rebuild." Compare pre-rebuild vs post-rebuild captures to the issue's expected behavior: - - Pre and post agree (artifact preserved) AND issue says it should be preserved → bug fixed - - Pre and post differ (artifact wiped) AND issue says it gets wiped → bug still reproduces - - Pre and post agree AND issue says it gets wiped → reproducer doesn't exercise the bug; Step 8c synth-repro - -The harness still uses Step 9's scoring framework — `+50 latest clean (artifact preserved)`, etc. — but the "what gets compared" axis is the diff, not the symptom phrase. - ---- - -## Step 8.5: Detect "Behavior Changed by Design" - -Before scoring, check whether the symptom is intentional. Some bugs are filed against behavior that was **deliberately changed or removed** in a merged PR — running the standard rubric on these produces misleading verdicts. The symptom "still reproduces" but the right answer is "won't fix, see PR #X." Issue #2791 is the prototype: `config set` was removed in PR #2227, the reporter tested a version that already had it gone, and a standard rubric run would have buried that context under a low-confidence `verify-inconclusive` label. - -This step is split into substeps so the rigor is mechanical, not optional. Every claim in the final comment must be backed by a verifiable evidence block — a comment URL with quoted phrase, a commit SHA with diff range, or a grep command with its actual output. Hand-wavy claims fail Step 8.5d's self-verification pass and force a bail to `verify-inconclusive`. - -### Step 8.5a: Run signal detection - -Any single signal is sufficient to trigger the by-design branch. - -**Signal 1 — Maintainer attribution in comments.** Any comment by an author with `authorAssociation` of `MEMBER`, `OWNER`, or `COLLABORATOR` matches `removed in #\d+`, `removed in [Pp][Rr] ?#\d+`, `by design`, `wontfix`, `won't fix`, `not a bug`, or `intentional`. - -```bash -gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ - --jq '.comments[] - | select(.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR") - | select(.body | test("removed in #\\d+|by design|wontfix|won.t fix|not a bug|intentional"; "i")) - | {url, author: .author.login, body}' -``` - -Capture for evidence: comment URL + author login + the exact quoted phrase. - -**Signal 2 — Removal commit in range.** A commit between the reported version and `$LATEST` deletes the symbol implicated by the reproducer (CLI subcommand, function, flag). The commit subject does NOT need to mention "remove" / "delete" — many removals ride into a `refactor(...)` or `feat(...)` commit (e.g. PR #2227 removed `--dangerously-skip-permissions` under a `refactor(sandbox): ...` subject). Use git's pickaxe to find the responsible commit by content: - -```bash -# Pickaxe: list every commit whose diff changes the count of <symbol> occurrences. -# Reverse order so the earliest removal commit lands first in the list. -git log "$REPORTED_VERSION".."$LATEST" -S'<symbol>' --reverse --oneline -- src/ bin/ nemoclaw/src/ - -# Subject-keyword narrowing is only a SUPPLEMENTARY lookup — useful when the -# pickaxe returns many commits and you want to focus on the obviously-removal one. -git log "$REPORTED_VERSION".."$LATEST" --grep='remove\|delete\|drop\|deprecate' -i --oneline - -# For each candidate, confirm the diff actually deletes the symbol (not just renames or moves it). -git log -p <candidate-sha> -- src/ bin/ nemoclaw/src/ | grep -nE '^-.*\b<symbol>\b' -``` - -Capture for evidence: commit SHA + each `file:line` block of deletions touching the symbol. Note the commit's actual subject — don't assume it says "remove." - -**Signal 3 — Symbol absent in both reported version and latest.** The implicated symbol (e.g. `config set`) is not present in either tag's source tree — meaning the responsible change landed before the version the reporter tested. This is the #2791 case. - -```bash -git grep -n "<symbol>" "$REPORTED_VERSION" -- src/ bin/ nemoclaw/ # expect: zero matches (or shim-only — see sub-case) -git grep -n "<symbol>" "$LATEST" -- src/ bin/ nemoclaw/ # expect: zero matches (or shim-only) -``` - -Capture for evidence: both grep commands and their (empty) outputs. - -**Sub-case for signals 2 and 3 — vestigial deprecation shims.** It's common for a removed symbol to survive in latest *only* as a deprecation message (e.g., a CLI subcommand that prints `"--<flag> was removed; use <X> instead"` and exits non-zero). When a grep returns matches in latest, inspect each `file:line`. If every match is a deprecation stub with no functional effect on the bug-as-filed, signal 2 or 3 still fires; record the shim locations and behavior as a separate evidence block. Do not silently treat shims as functional code, and do not silently treat them as absence. - -### Step 8.5b: Pre-check related failure modes - -A by-design verdict says "the bug *as filed* can't reproduce." It does NOT say "every bug shaped like this is fixed." Before drafting the comment, search latest's source for code paths that could still produce the issue's described **symptom** (not the literal removed flag/symbol — the symptom). - -```bash -# Use the issue's symptom keywords, not the removed symbol. -git grep -nE "<symptom-keyword-1>|<symptom-keyword-2>" "$LATEST" -- src/ nemoclaw/src/ -``` - -For #2168 the literal flag is `--dangerously-skip-permissions`, but the symptom is "sandbox created but not registered in CLI." Grepping for `register.*[Ss]andbox`, the readiness-gate / cleanup-failure path in `src/lib/onboard.ts` surfaces as a related-but-different way to produce an orphan sandbox. - -If a related failure mode is found, the by-design comment MUST include a "What's not literally the same bug" section that names it with `file:line`. Don't suppress the call-out by claiming "the symptom is impossible" when the symptom can be reached via a different path. - -### Step 8.5c: Check existing test coverage - -Search the repo for tests that exercise the NEW intended workflow (the one that replaced the removed symbol). Citing them strengthens the comment from "trust me, it was removed" to "the new workflow is exercised by these tests." - -```bash -git grep -lnE "<new-workflow-keyword>" -- test/ nemoclaw/src/ 2>/dev/null | head -5 -``` - -Cite at most three concrete test paths. If none exist, omit the section — do not invent paths. - -### Step 8.5d: Self-verification pass before posting - -Two passes, both required. - -**Evidence pass.** Re-run every grep / git / `gh` command cited in the evidence blocks. If any cited `file:line`, commit SHA, or quoted output doesn't reproduce on a fresh invocation, **stop and revise** — or bail to `verify-inconclusive` if the discrepancy can't be resolved. - -**Link pass.** Resolve at least one rendered markdown link from each section that has them — `What's structurally fixed`, `Vestigial references`, `Existing CI coverage`. Use `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 content if the path exists at the tag, 404 otherwise) or `curl -fsI <blob-url>` (returns 200 if the blob renders). A broken link is worse than no link — it suggests verification work that didn't actually happen. - -The cost of an incorrect "I checked and X is gone" claim in a public comment, or a 404 on a citation, is higher than spending a minute re-checking. This step exists because LLMs can confidently overstate and confidently invent paths; mechanical re-verification catches both. - -### Step 8.5e: If any signal fires - -- **Skip the Step 9 score table** entirely. The "exit 0 + expected output" axis doesn't apply when the expected output is no longer the contract. -- **Skip Brev provisioning** if the signal fires before Step 7 — a remote run would just confirm what static analysis already proved. (Signals 2 and 3 can run as soon as the reported version is parsed in Step 4.) -- **Apply label `status: wont-fix`** (the existing repo label — quote it on the CLI: `gh issue edit <num> --add-label "status: wont-fix"`). It's already in the Step 3 issue-type skip list, so a labelled issue is automatically excluded from future runs without needing a separate idempotency clause. -- **Use the by-design comment template below** instead of the standard Step 10 template. -- **@-mention the reporter** so they can object if the framing is wrong. -- **Never auto-close.** A maintainer pulls the trigger, same as the other label paths. - -### By-design comment template - -Mandatory sections in this order. Omit only the sections explicitly noted as omittable. - -**Tag-anchoring + linking rule.** Every `file:line` citation, commit SHA, and test-path reference in the rendered comment MUST be a clickable markdown link to the verified-on tag (e.g., `v0.0.35`), not the maintainer's working `HEAD`. Lines drift between tags and main; tag-anchored links keep the citations reproducible by anyone reading the comment months later. Bare paths force the reader to navigate manually — that's a usability bug, not a stylistic preference. - -Use these exact link formats: - -- File only: `[src/lib/onboard.ts](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/onboard.ts)` -- File:line: `[src/lib/onboard.ts:4965](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/onboard.ts#L4965)` -- File:line-range: `[src/lib/commands/sandbox/connect.ts:25-31](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/src/lib/commands/sandbox/connect.ts#L25-L31)` -- Commit SHA: `[5956a61](https://github.com/NVIDIA/NemoClaw/commit/5956a612e18047b9ab85b3a7e89f6b5dedb29190)` — short SHA as the link text, full SHA in the URL -- Test file: `[test/e2e/test-double-onboard.sh](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/test/e2e/test-double-onboard.sh)` -- PR/issue references: bare `#NNNN` works — GitHub auto-links these in comments on the same repo, no manual URL needed. - -When greping for evidence, use `git grep -n "<symbol>" "$LATEST" -- ...` so the line numbers match the tagged blob. Then construct each link from `<file path> + verified-on tag + line number`. - -The Step 8.5d self-verification pass MUST resolve at least one rendered link (e.g., `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=v0.0.35` or a `curl -fsI` to the blob URL) and confirm it returns the expected file. A broken link defeats the purpose of including the citation. If any link fails to resolve, fix it or bail to `verify-inconclusive`. - -````markdown -## Stale-issue verification — behavior is by-design - -**Reported on:** v0.0.<X> -**Verified on:** v0.0.<Y> (PR #<NNNN> first shipped in v0.0.<Z>) -**Verification mode:** static analysis at the verified-on tag — no runtime reproduction. Step 8.5 by-design short-circuits Brev provisioning because the responsible code change is already proven by the diff between `$REPORTED_VERSION` and `$LATEST`. -**Outcome:** symptom reproduces against the reproducer as filed, but the implicated behavior was intentionally changed. - -### What's structurally fixed - -- `<file:line>` — `<one-sentence summary of the change at that location>` -- `<file:line>` — `<…>` - -The new workflow is `<one-sentence: how to do what the user was trying to do>`. - -### Vestigial references - -- `<file:line>` — `<deprecation behavior: e.g. "prints '--<flag> was removed; use <X> instead' and exits 1; no functional effect">` - -(Omit this section entirely when the symbol is fully gone with no surviving stubs.) - -### What's not literally the same bug - -`<one-sentence acknowledgement of the related failure mode found in Step 8.5b, with file:line>` — OR — `None. The symptom requires the removed symbol; no related code path produces it on latest.` - -### Existing CI coverage - -- `<test/path/file>` — `<one-sentence: what this test demonstrates about the new workflow>` - -(Omit when no direct test exists. Do not invent paths.) - -### Recommendation - -@<reporter> — please confirm the by-design framing is correct (the implicated `<symbol>` was intentionally removed, the original reproducer can no longer execute) and close as "won't fix / by design" if you agree. If a related symptom (e.g. `<related failure mode from above>`) is hitting you on ≥ v0.0.<Z>, please file a fresh issue with a v0.0.<Z>+ reproducer. - -`<NVBugs cross-ref line — see below>` - -<!-- nemoclaw-verify-stale v1 YYYY-MM-DD --> -```` - -**NVBugs cross-ref line.** If `NVBUGS_REF` was set in Step 4, append: - -> NVBugs<NVBUGS_REF without brackets> will need a separate update; closing this GitHub issue won't propagate. - -Otherwise omit the sentence. - -**If no signal fires:** continue to Step 9 normally. - ---- - -## Step 9: Score Confidence - -Start at 0. Apply each rule that fires. - -| Signal | Delta | -|---|---| -| Reproducer ran cleanly on **latest** (8d), exit 0, no bug symptom observed | +50 | -| Commits between reported version and `$LATEST` touch the implicated component (see "Path extraction" below) | +25 | -| A merged PR mentions this issue number or its symptom (see "PR search" below) | +25 | -| Reproducer was LLM-synthesized at any point (Step 8b synth or Step 8c retry) | −30 | -| Any partial error, warning, or flaky behavior in the latest run (8d) | −50 | - -Total is clamped to `[0, 100]`. - -### Path extraction (for the +25 commits signal) - -The skill needs to know *which* path to `git log v<reported>..$LATEST -- <path>` against. Apply in order, stop at the first that yields a non-empty path: - -1. **Stack trace / file path mentions in the issue body.** Grep the body for absolute paths under known install roots, then map to repo paths: - - `/usr/local/lib/nemoclaw/<rel>` → `<rel>` in repo (e.g., `scripts/generate-openclaw-config.py`) - - `/usr/local/bin/nemoclaw*` → `bin/` - - `~/.nemoclaw/<rel>` → most often runtime state, drop unless the bug is config-related → `src/lib/config/` - - In-repo paths (e.g., `bin/lib/policies.js` mentioned literally) → use as-is -2. **Component-label-to-directory map.** Pick the first match. Paths verified against the current repo layout — drop any path that doesn't exist on the tag at `$LATEST` rather than passing it to `git log`. - - `NemoClaw CLI` → `bin/`, `src/lib/`, `nemoclaw/src/commands/` - - `Sandbox` → `nemoclaw/src/blueprint/`, `nemoclaw-blueprint/` - - `OpenShell` → cross-repo (lives at `github.com/NVIDIA/OpenShell`, not in this repo). Skip the +25 signal for OpenShell-only issues; cross-repo `git log` is out of v1 scope. - - `Docker` → `Dockerfile`, `Dockerfile.base`, `scripts/install-openshell.sh`, `scripts/install.sh` - - `Getting Started` → `docs/`, `scripts/install.sh` - - `Integration: <X>` — no `src/lib/integrations/` exists in this repo. Skip the +25 signal for integration-component issues unless source 1 (file paths in body) yielded a path. -3. **Title keywords.** "policy" → `nemoclaw-blueprint/policies/`, `nemoclaw/src/blueprint/`. "inference" → `docs/inference/` is docs-only; skip the +25 signal unless source 1 surfaces actual code paths. - -If none of the above produces a path, **skip the +25 signal entirely** rather than guessing. Floating the +25 on every issue would inflate scores meaninglessly. - -### PR search (for the +25 PR signal) - -```bash -# Direct issue-number reference (covers most cases — "fixes #2861" etc.) -DIRECT_REF=$(gh pr list --repo NVIDIA/NemoClaw --state merged \ - --search "$ISSUE_NUMBER" \ - --json number,title,mergedAt,body \ - -q "[.[] | select((.body + \" \" + .title) | test(\"#$ISSUE_NUMBER\\\\b\"))]") - -# Symptom-phrase fallback (only if direct reference returns nothing) -if [ -z "$DIRECT_REF" ] || [ "$DIRECT_REF" = "[]" ]; then - SYMPTOM=$(extract first key error/symptom phrase from issue body, ~3-6 words) - SYMPTOM_REF=$(gh pr list --repo NVIDIA/NemoClaw --state merged \ - --search "\"$SYMPTOM\"" \ - --json number,title,mergedAt) -fi -``` - -Apply +25 if either query returns at least one PR with `mergedAt` strictly after the tag date of `$REPORTED_VERSION` (look up via `git log -1 --format=%cI v$REPORTED_VERSION`). PRs merged before the reporter even filed the issue can't have fixed it. - -If neither query returns anything, **skip the +25 signal**. - -**Baseline-validation gating.** The +50 weight assumes the reproducer was *validated* — i.e., it produced the bug symptom on baseline (Step 8b/8c match). If `BASELINE_INSTALL_FAILED=1` (Step 8a fall-through, baseline pass skipped — including the sandbox-build-rot case from Step 11), the +50 still applies but **cap the total at 84**. Corroboration signals (commits-touched-area, PR-mention) still raise the score within the cap but cannot lift it above 84. Without runtime baseline confirmation we don't have enough on our own to claim ≥85 — the cap forces the verdict into the 60–84 band where the reporter is asked to confirm. The previous draft of this rule had an "unless commits-touched OR PR-mention also fires" escape hatch that let inferred fix evidence bypass the cap entirely; that produced a misleading 100/100 on the #2007 e2e run despite zero baseline confirmation, and was tightened here. - -**Action (when latest run was clean — bug not reproduced):** - -| Score | Label | Comment | -|---|---|---| -| ≥85 | `fixed-on-latest` | Evidence-rich, no @-mention. | -| 60–84 | `fixed-on-latest` | Evidence-rich, **@-mention the original reporter** to confirm. | -| <60 | `verify-inconclusive` | Short, honest "couldn't verify" explanation. | - -**Special case: latest output matches the issue symptom (bug still reproduces on latest).** - -This is not a flake — the skill positively confirmed the bug is still live. Don't apply the +50 weight (the bug isn't fixed) and skip the score table entirely. - -- Post a "still reproduces on latest" comment with both transcripts. -- Apply **no label**. -- Include the marker `<!-- nemoclaw-verify-stale v1 YYYY-MM-DD -->` with today's date so the candidate filter applies the 7-day TTL (Step 3 idempotency). -- Next weekly run picks the issue back up after the TTL — if the bug gets fixed in the meantime, that run catches it. - -The skill **never closes issues** in any branch. A maintainer pulls that trigger after reviewing the label and comment. - ---- - -## Step 10: Compose and Post the Comment - -**Redaction pass before posting.** Run on **every** chunk of text quoted in the comment — issue body excerpts, baseline transcript, latest transcript, synth-repro scripts. Replace each match with `[REDACTED]`. The transcripts especially leak — they include full stdout/stderr from real installs and runs. - -**HTML → text pre-pass for issue body excerpts.** NV QA bodies are HTML; tokens nested in `<pre>` tags or HTML attributes (e.g. `<a href="https://user:tok@host/...">`) slip past the regex patterns below if the input still has tags. Convert to plain text first, then redact: - -```bash -TEXT=$(printf '%s' "$BODY_EXCERPT" | python3 -c ' -import html, re, sys -b = sys.stdin.read() -b = re.sub(r"<br\s*/?>", "\n", b) -b = re.sub(r"</?(p|div|tr|td|th|li|pre)[^>]*>", "\n", b) -b = re.sub(r"<[^>]+>", "", b) -print(html.unescape(b)) -') -# Now apply the regex table below to $TEXT. -``` - -Transcripts and synth-repro scripts are already plain text and skip the pre-pass. - -**Order matters and the patterns below are in execution order.** Longest, most-specific patterns first; generic catchalls last. Otherwise the catchall masks specific matches and you lose track of what was actually redacted (JWT vs session blob vs random base64). - -Patterns live in a fenced block (not a markdown table) because patterns 8 and 9 use regex alternation `|` — markdown tables would treat the literal `|` as a column delimiter, and escaping it as `\|` makes the regex match a literal pipe instead of an alternation, which silently breaks credential redaction. - -```regex -1. eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,} - → JWT tokens - -2. gh[pousr]_[A-Za-z0-9]{36,} - → GitHub PATs / install tokens - -3. (?i)nvapi-[A-Za-z0-9_-]{20,} - → NVIDIA API keys (NIM / build.nvidia.com) - -4. AKIA[0-9A-Z]{16} - → AWS access key IDs - -5. (?i)aws_secret_access_key\s*=\s*\S+ - → AWS secret keys - -6. (?i)authorization:\s*\S+ - → HTTP auth headers (often Bearer + JWT) - -7. URLs containing `@` before the host (e.g., https://user:pw@host/...) - → Basic-auth credentials in URLs - -8. (?i)(token|secret|password|api[_-]?key|bearer)[^\n]*[:=][^\n]* - → Inline credentials in env/config/log output - -9. \b\w+\.(nvidia\.internal|nv-internal\.com|nvidia\.dev)\b - → Internal hostnames (extend list per team) - -10. [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} - → Email addresses (PII) - -11. \b[A-Za-z0-9+/]{60,}={0,2}\b - → Long base64 blobs (likely keys/sessions; tune length to taste — too short hits legit data) -``` - -**File paths under the reporter's home directory** (`/Users/<name>/`, `/home/<name>/`) → replace with `~/`. Run last; catches incidental username PII. - -**Comment authoring principle.** Every section in a rendered comment must either change a reader's mind about the verdict, or be cut. Word counts follow from that — **300 is a hard ceiling** for the main verdicts (fixed-on-latest, wontfix). Simple cases (clear PR ref, deterministic check) land under 200. The principle generalizes: comments posted by this skill compete for a maintainer's attention against every other in-flight thread, and "AI-slop" prose — architectural sidebars, file:line citations the maintainer can find via the PR ref, bare-output reproductions when the load-bearing evidence is elsewhere, "if this verification is wrong, please reopen…" boilerplate — actively reduces the comment's signal-to-noise ratio. - -**For each section in a draft, ask: would the maintainer reach a different conclusion *without* this section? If no, delete.** Lessons accumulated from real runs: - -- **#2007 first draft (~750 words):** had a multi-paragraph "Architectural notes for QA reference" section that didn't change the verdict. Cut → 371 words. -- **#2604 first three drafts:** wavered between fixed-on-latest, still-reproduces, and by-design across iterations because each draft padded the verdict with prose that didn't ground it. Final 190-word draft cut a maintainer-note sidebar about platform attribution, a bare-status output reproduction, and a file:line citation of the source — none affected the verdict, all were AI-slop padding. Rule learned: **before drafting any prose, name the verdict in one sentence; if a section doesn't directly support that one sentence, cut it before writing it.** - -**Per-verdict length defaults:** - -| Verdict | Target | Rationale | -|---|---|---| -| `fixed-on-latest` | **200–300 words** | Header + evidence + verdict + @-mention. Add hardware-substitution caveat or related-failure-mode section only if they shift the maintainer's read. If you're past 300, you're padding. | -| `wontfix` (by-design) | **200–300 words** | Structurally-fixed + vestigial + what's-not-the-same-bug, each one to two sentences max. The PR ref carries the detail; the comment carries the verdict. | -| `verify-inconclusive` | 100–200 words | One paragraph naming what the skill couldn't establish. No transcripts beyond a single quoted line. | -| **Still-reproduces (no label)** | **30–80 words** | The reporter already has the symptom; the maintainer can see the issue is open. The skill is just confirming + setting the TTL marker. **No transcripts** (the issue body has them), **no @-mention** (the reporter knows their bug is real), **no architectural prose**. One sentence stating "skill ran reproducer on `<latest>`, symptom still present" + one sentence on any partial-fix PR if relevant + marker. That's it. | - -**Cut, by default:** - -- Maintainer-note sidebars about labels / platform attribution unrelated to the bug surface. -- Bare-output reproductions when the load-bearing evidence is in a different command's output. -- File:line citations of source code already findable via the cited PR. -- Closing "if this verification is wrong, please reopen…" boilerplate. -- Redundant verbal framing of what the evidence already shows ("the table above proves…"). -- "Verification mode" pleasantries beyond one factual line. - -**Mandatory cap caveat.** When the score is capped (Step 9 baseline-validation gating, or any Step 11 degraded-mode path), the rendered Verdict section must include a one-line caveat naming the cap and the reason. Example: `Capped at 84 because Step 9's baseline-validation gate did not run (sandbox-build rot on v0.0.18: Dockerfile symlink layer removed by #2227).` Don't make readers reverse-engineer why the score didn't go higher — name it. - -**Mandatory hardware-substitution caveat.** When the issue carries `Platform: DGX Spark` or `Platform: GB10` and Step 7 provisioned a Brev SKU that is not the same silicon (Brev's stoppable GPU catalog is x86 + discrete H100/A100/L40S/T4 — not Grace Hopper / GB10 unified-memory ARM64), the rendered comment must include a one-line "Hardware substitution" note. Example: `Hardware substitution: verified on Brev n1-standard-4:nvidia-tesla-t4 (x86_64 + T4) as a substitute for the reporter's DGX Spark (ARM64 + GB10). For silicon-shape bugs (perf, memory architecture, drivers) this is not a faithful repro — please confirm on actual DGX Spark.` This goes in the metadata block right after `Verification mode:` so it's visible at the top, not buried in the analysis. - -**Mandatory `Verification mode` header line.** All three templates below include a `**Verification mode:**` line in the metadata block, naming what we did and didn't actually run (e.g., "runtime reproduction on Brev <SKU>; baseline + latest both installed and run" for the standard template; "static analysis at the verified-on tag — no runtime reproduction" for the by-design template; "runtime reproduction on Brev <SKU>; bug confirmed live on latest" for still-reproduces). Reader should never have to guess whether the verdict came from real install logs or from static analysis. - -**Link-pass self-verification (all templates).** Same rule as Step 8.5d's link pass, applied to every template. Resolve at least one rendered markdown link from each section that has them (`What's structurally fixed` / `Vestigial references` / `Existing CI coverage` for by-design; `Relevant changes since` / transcript code-anchor citations for the standard template) via `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 if path exists at tag, 404 otherwise) or `curl -fsI <blob-url>`. A 404 on a citation in the rendered comment is worse than no citation — it advertises verification work that didn't actually happen. If any link fails to resolve, fix it or bail to `verify-inconclusive`. - -**Mandatory closing block — reporter @-mention with confirmation language.** Every template below **except `Still-reproduces`** ends with an explicit @-mention of the original reporter using this exact shape: - -> @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. - -The skill cannot independently confirm a closed-as-fixed verdict — only the reporter knows whether their original symptom is gone in their environment. The @-mention is what converts a "skill says it's fixed" claim into actionable confirmation work for QA. Customize `<Z>` per case (the version that shipped the fix or `$LATEST`), but never omit the line. - -**Mandatory unanswered-question prefix and dual @-mention.** When Step 3 sets `UNANSWERED_MAINT_LOGIN` (a maintainer's question is older than 7 days and the reporter never replied), the verdict comment changes shape in two places: - -1. **Prepend a lead paragraph** as the very first line of the body, before the `## Stale-issue verification` heading. The lead paragraph is a single line: - - ```text - [@UNANSWERED_MAINT_LOGIN's comment](UNANSWERED_MAINT_URL) from UNANSWERED_MAINT_DATE is still unanswered. Posting independent verification below to unstick the thread. - ``` - - …with the bracketed variables expanded from the values exported by Step 3. - -2. **Replace the closing reporter-only @-mention with a dual @-mention** that names BOTH the maintainer (acknowledging the open question) and the reporter (per the standard confirmation pattern): - - > @\<UNANSWERED_MAINT_LOGIN\> — flagging that your question above is still open; the verification below may answer it. @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. - -This applies to all three templates (fixed, still-reproduces, by-design). The skill becomes the *unsticking voice* on a thread that has gone quiet — never a clueless interruption when discussion is fresh (Step 3 already filtered the within-7-day case). - -**Comment template (fixed / inconclusive — bug not reproduced on latest):** - -````markdown -## Stale-issue verification — automated - -**Reported on:** v0.0.31 -**Verified on:** v0.0.34 (commit abc1234) -**Verification mode:** runtime reproduction on Brev `<instance-class>` — baseline (v0.0.31) and latest (v0.0.34) both installed and run; comparison made on the captured transcripts. (Or: "runtime reproduction on Brev `<instance-class>` — baseline-install-skipped (`.openclaw-data` rot, see Step 11), latest-only run; verdict capped at 84.") -**Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 / <CUDA version if GPU> - -### Baseline (reported version) - -- Install: succeeded · skipped (install rotted) -- Reproducer: extracted verbatim · synthesized (−30 penalty) -- Result: bug symptom matched (validated) · could not validate (skipped Step 8c gate) - -<details><summary>Baseline transcript</summary> - -```text -<full baseline transcript> -``` - -</details> - -### Latest - -- Install: succeeded -- Result: not reproducible — clean run, no bug symptom observed - -<details><summary>Latest transcript</summary> - -```text -<full latest transcript> -``` - -</details> - -### Verdict - -**Confidence:** 88 / 100. Labelling `fixed-on-latest`. - -<details><summary>Relevant changes since v0.0.31</summary> - -- abc1234 — fix: <commit subject> -- def5678 — refactor: <commit subject> - -</details> - -@<reporter> — please confirm the symptom is gone on a recent build (≥ v0.0.<Z>) and reopen with a fresh reproducer if you observe otherwise. - -<!-- nemoclaw-verify-stale v1 2026-05-12 --> -```` - -**Comment template (still reproduces — Step 9 special case):** - -````markdown -## Stale-issue verification — still reproducible - -**Reported on:** v0.0.31 -**Verified on:** v0.0.34 (commit abc1234) -**Verification mode:** runtime reproduction on Brev `<instance-class>` — baseline confirmed the symptom matches the issue; latest (v0.0.34) also produced the symptom. Bug is still live. -**Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 - -The skill ran the reported reproducer on v0.0.34 and observed the same bug symptom described in this issue. The bug is still live. - -No label applied. Will re-verify automatically next weekly run; if a fix lands in the interim, the next pass catches it. - -@<reporter> — please confirm the symptom still matches your observation on v0.0.<Y> and reopen with any updated reproducer or environment details if it has shifted. - -<details><summary>Baseline transcript (validated reproducer)</summary> - -```text -<baseline transcript> -``` - -</details> - -<details><summary>Latest transcript (bug still observed)</summary> - -```text -<latest transcript> -``` - -</details> - -<!-- nemoclaw-verify-stale v1 2026-05-12 --> -```` - -The trailing HTML comment is the **idempotency marker** Step 3 looks for. Always include today's date in `YYYY-MM-DD` format so the candidate filter can apply the 7-day TTL. - -**Pre-post state-check.** A long-running verification can race with the maintainer closing the issue independently — happened on #2513 and #2519 (mid-batch closes by @jyaunches with their own verification). Re-check `state == OPEN` right before posting. If closed, apply the label tag-only (skipping the comment, since the maintainer's own close-comment is now the authoritative record) and skip the Project 199 move. - -```bash -STATE=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json state --jq .state) -if [ "$STATE" != "OPEN" ]; then - echo "[verify-stale] #$ISSUE_NUMBER closed since verification started — applying label tag-only, skipping comment + tracker move" - gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "$LABEL" - exit 0 -fi -``` - -**Post the comment and apply the label:** - -```bash -gh issue comment "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --body-file comment.md -gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "fixed-on-latest" -# or for <60: -# gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "verify-inconclusive" -``` - -**Move the issue to "Needs Review" on the NemoClaw Development Tracker AND self-assign (only on `fixed-on-latest`).** The tracker is GitHub Project [NVIDIA/199](https://github.com/orgs/NVIDIA/projects/199) ("NemoClaw Development Tracker"). When the skill's verdict is `fixed-on-latest`, the issue moves to **Needs Review** AND the issue is assigned to the maintainer who ran the skill (`$GH_IDENTITY` from Step 6.5) — assignment puts the issue in their personal review queue so they don't lose track of what they've staked their name on. After the reporter confirms and the maintainer closes, existing Project automation (or a manual move) advances it to Done. **No move and no assign on `wontfix` / `verify-inconclusive` / no-label-still-reproduces** — those have separate close paths. - -This step requires the `project` scope on the maintainer's gh CLI (`gh auth refresh -h github.com -s project` in a real terminal once; OAuth device-code flow). If the scope is missing, the lookup query returns an auth error — fall through with a one-line warning rather than failing the whole run. - -```bash -# Project 199 constants (re-run gh project field-list 199 --owner NVIDIA --format json -# if the project gets renamed/restructured and these IDs drift): -PROJECT_ID="PVT_kwDOABpemM4BSCP5" -STATUS_FIELD_ID="PVTSSF_lADOABpemM4BSCP5zg_r9p8" -NEEDS_REVIEW_OPTION_ID="5c5922a9" - -# Only fire on fixed-on-latest. Skip silently otherwise. -if [ "$VERDICT" = "fixed-on-latest" ]; then - # Find the issue's existing project item, if any. - ITEM_ID=$(gh api graphql -f query=' - query($num: Int!) { - repository(owner: "NVIDIA", name: "NemoClaw") { - issue(number: $num) { - projectItems(first: 10) { - nodes { id project { number } } - } - } - } - }' -F num="$ISSUE_NUMBER" \ - --jq '.data.repository.issue.projectItems.nodes[] | select(.project.number == 199) | .id' \ - 2>/dev/null | head -1) - - # If the issue isn't on the project yet, add it. (NV QA bots usually add new - # issues automatically, but cover the gap.) - if [ -z "$ITEM_ID" ]; then - ITEM_ID=$(gh project item-add 199 --owner NVIDIA \ - --url "https://github.com/NVIDIA/NemoClaw/issues/$ISSUE_NUMBER" \ - --format json --jq .id 2>/dev/null) - fi - - if [ -n "$ITEM_ID" ]; then - gh project item-edit \ - --id "$ITEM_ID" \ - --project-id "$PROJECT_ID" \ - --field-id "$STATUS_FIELD_ID" \ - --single-select-option-id "$NEEDS_REVIEW_OPTION_ID" \ - >/dev/null && echo "[verify-stale] moved #$ISSUE_NUMBER to 'Needs Review' on Project 199" - else - echo "[verify-stale] WARN could not resolve project item for #$ISSUE_NUMBER on Project 199 — label applied but tracker not moved" - fi - - # Self-assign the issue to the maintainer who ran the skill — puts it in their - # personal review queue alongside the Needs Review state. - gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-assignee "$GH_IDENTITY" \ - >/dev/null && echo "[verify-stale] assigned #$ISSUE_NUMBER to @$GH_IDENTITY" -fi -``` - -The Step 12 activity log line should record the project move (or the warn-and-skip case) so a maintainer scanning the log can spot tracker drift. Add a `Tracker:` row to the per-issue entry: `Tracker: moved to Needs Review` | `not moved (verdict: <X>)` | `not moved (project lookup failed)`. - ---- - -## Step 11: Infra Failure Handling - -Two different failure types, two different responses. - -**Latest-install failure** (Step 8d) or reuse-check / provisioning / harness errors: hard infra failure. - -- Print the error. -- Apply **no label** — infra failures must not pollute the verification record. -- Post a short comment **only if explicitly requested by the invoking user**. Default is silent move-on. -- Continue to the next candidate in batch mode. - -The next weekly run retries naturally. - -**Baseline-install failure** (Step 8a, reported version won't install on a modern image): not a hard failure — degraded mode. - -- Set `BASELINE_INSTALL_FAILED=1`, skip 8b/8c, jump to 8d. -- Step 9 applies the score cap (max 84) — corroboration signals raise the score within the cap but cannot lift past it. -- Note "baseline-install-skipped" in the final comment so a reviewer knows the verification ran without the script-validation gate. - -**Baseline-build failure** (Step 8a binary install succeeded, but the in-image `Dockerfile` build during sandbox creation failed on a layer that was structurally removed in a later release): also degraded mode, distinct from binary install rot. Surfaced during the #2007 e2e run on v0.0.18 (`/sandbox/.openclaw-data/workspace/media` symlink layer, removed entirely by #2227). - -- Set `BASELINE_INSTALL_FAILED=1` (same flag — Step 9's cap-at-84 rule keys off it regardless of which phase rotted). -- Skip 8b/8c, jump to 8d. -- Note "baseline-build-skipped" in the final comment with the specific failing layer/file so a reviewer can see *why* the v0.0.X image no longer builds (the why is usually a follow-on PR that removed the rotted layer). -- Do not retry the build with a patched Dockerfile — that breaks faithfulness. We're claiming "couldn't independently re-trigger the original symptom on baseline," not "we made the old version work somehow." - -Both baseline-rot variants share the same downstream effect: Step 9 cap, Step 10 caveat, @-mention reporter to confirm. Distinguishing them in the comment helps a reviewer understand the failure mode without re-running. - -This degradation is expected — old releases rot at multiple phases (binary installer URL drift, base-image dependencies vanish, in-image Dockerfile layers get removed by structural refactors). We still want to extract whatever signal we can from the latest run plus PR/commit evidence, just at a more conservative confidence ceiling. - -**Empirical reality after two e2e runs:** baseline-build-rot is the **dominant** failure mode for any reported version more than ~5–7 patches behind, not an edge case. Both #2007 (v0.0.18, 17 patches behind) and #2592 (v0.0.28, 7 patches behind) hit it. The cap-at-84 with reporter @-mention is the **modal** verdict shape for stale-issue verification, not the exception. Reframe expectations accordingly: - -- For issues reported >5 patches behind `$LATEST`, plan for the cap-at-84 path. Pre-flight (PR-search, pickaxe) carries more weight than baseline runtime evidence. -- For issues reported within 1–4 patches of `$LATEST`, baseline is more likely to install cleanly and the full +50 path is reachable. -- The skill's design assumes baseline + latest both run cleanly; in practice latest-only with cap-at-84 is the workhorse path. The score-cap is doing real work, not just a fallback. - -**Keep-box-on-inconclusive.** When `verify-inconclusive` lands (Step 8c gave up, or Step 9 score < 60), **skip the cleanup trap** for this run if the box was provisioned by this run — set `PROVISIONED_NEW=0` before the trap fires so the EXIT handler is a no-op. Print the `brev shell "$INSTANCE_NAME"` command and an explicit `brev delete "$INSTANCE_NAME"` reminder in the run output so the maintainer can triage and clean up manually. Reused boxes stay regardless. Ship-failed verifications are the exact case where having an inspectable artifact pays for itself; an unbounded sleep-and-delete in the background isn't reliable across session ends, so we leave deletion explicit. - ---- - -## Step 12: Log to Activity - -After each issue (verified, inconclusive, by-design, or infra-failed), append to `${VERIFY_STALE_LOG_DIR:-$HOME/development/daily-rhythm/activity}/nemoclaw-verify-stale-log.md`. The default path matches the personal-organizer convention; export `VERIFY_STALE_LOG_DIR` to point elsewhere (CI, shared volume, etc.). Create the directory if missing — do not assume it exists. - -```markdown -### NVIDIA/NemoClaw#<number> — <title> -**Date:** YYYY-MM-DD -**Reported on:** v0.0.31 -**Verified on:** v0.0.34 -**Environment:** CPU | GPU (<instance type>) -**Box:** reused <name> | provisioned <name> | local (no Brev — Step 6.7 short-circuit) -**Baseline install:** succeeded | failed (degraded mode) -**Baseline match:** validated (verbatim) | validated (synth) | failed (verify-inconclusive) | skipped -**Latest install:** succeeded | failed (infra error) -**Latest result:** not-reproduced (clean) | still-reproduces | partial / flake | n/a (skipped 8d) -**Confidence:** 88 / 100 | n/a (still-reproduces) -**Label applied:** fixed-on-latest | verify-inconclusive | status: wont-fix | none (still-reproduces) | none (infra) -**Tracker:** moved to Needs Review on Project 199 | not moved (verdict: <X>) | not moved (project lookup failed) -**Assignee:** @<GH_IDENTITY> | not assigned (verdict: <X>) -**Brev wall time (approx):** N min - ---- -``` - -Create the file if missing, with this header: - -```markdown -# NemoClaw — Verify Stale Log - -A running record of stale-issue verification runs on NVIDIA/NemoClaw. -Persisted via daily-rhythm to GitLab. - ---- -``` - -At end of a batch session, prepend a session summary: - -```markdown -## YYYY-MM-DD — Verify Session -**Issues considered:** N -**Verified `fixed-on-latest`:** N -**Marked `status: wont-fix` (by-design path):** N -**Marked `verify-inconclusive`:** N -**Local-first short-circuits (no Brev cost):** N -**Skipped (Windows / macOS / integration / no version):** N -**Infra failures:** N -**Brev wall time:** N min · approx $X.XX - ---- -``` - -Never stage or commit the log to the NemoClaw repo. - ---- - -## Cadence - -- **Weekly cron** — Monday morning, batch mode, ≤15 issues (the Step 1 cap, sliced after Step 3/4 filters). -- **Manual** — invoke with a single issue number anytime. - ---- - -## Out of Scope (v1) - -- Auto-closing issues. Always tag-only; a human pulls the trigger. -- macOS verification *via the Brev path*. Brev offers no macOS instances. The Step 6.7 local-first short-circuit *does* run on a maintainer's macOS laptop — so manual single-issue runs against pure-CLI bugs work on macOS. The weekly batch cron is Linux-only because that path always uses Brev. -- Issues requiring third-party integration credentials (Slack, Discord, Telegram, Hermes, OpenClaw, WeChat). -- Service-account bot identity. v1 runs under each maintainer's own GitHub credentials. -- Versioned labels. A single `fixed-on-latest` label is swept on each release cut. - ---- - -## Companion Behavior - -`nemoclaw-maintainer-cut-release-tag` sweeps `fixed-on-latest` and `verify-inconclusive` from all open issues at release time. Without that sweep, "latest" drifts and verifications go stale silently. The by-design path uses the existing repo `status: wont-fix` label; that label is **not** swept (it's also applied for non-skill reasons such as scope or priority decisions, and clearing it would erase human triage work). diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/reproduction-rubrics.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/reproduction-rubrics.md new file mode 100644 index 00000000000..0bb2acc8fa8 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/reproduction-rubrics.md @@ -0,0 +1,248 @@ +<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> +<!-- SPDX-License-Identifier: Apache-2.0 --> + +# verify-stale — Reproduction Rubrics Reference + +Use after the baseline/latest installs are ready. Covers baseline matching, synth-repro retry, latest rerun, architectural drift, performance bugs, and rebuild-cycle bugs. + +## Contents + +- [Step 8b: Run reproducer on baseline, compare to issue symptom](#step-8b-run-reproducer-on-baseline-compare-to-issue-symptom) +- [Step 8c: Synth-repro and retry on baseline](#step-8c-synth-repro-and-retry-on-baseline) +- [Step 8d: Install latest, run validated reproducer](#step-8d-install-latest-run-validated-reproducer) +- [Step 8d.5: Architectural-Drift Check](#step-8d5-architectural-drift-check) +- [Step 8e: Performance-Bug Verification](#step-8e-performance-bug-verification-when-bug_classperformance) +- [Step 8f: Rebuild-Cycle Verification](#step-8f-rebuild-cycle-verification-when-bug_classrebuild-cycle) + +--- + +### Step 8b: Run reproducer on baseline, compare to issue symptom + +If `./reproducer.sh` exists (verbatim from Step 6), run it. Otherwise synth on demand from the issue body (apply −30 penalty now, locked in for the rest of the run). + +**Interactive subcommand handling.** Many `nemoclaw onboard` / `nemoclaw configure` invocations prompt for input and will hang in a non-interactive shell. Auto-detect such subcommands in the script and apply, in order: + +1. Add `--non-interactive` if the version supports it. +2. Add `--dangerously-skip-prompts` (issue #2168 confirmed this exists for at least some Jetson paths). +3. Pre-feed answers via stdin: `printf 'yes\n\n\n' | nemoclaw onboard ...` + +If none work, route the script to Step 8c (synth-repro) so the LLM can rewrite it using non-interactive equivalents. + +```bash +# `brev exec` spawns a non-login shell, so ~/.local/bin (where the nemoclaw binary lives +# after install) is not on PATH unless we export it. The reproducer script itself must +# use `sg docker -c '...'` blocks for any Docker-touching command — Step 8a.5b covers +# that requirement; double-wrapping with sg docker on the outer call breaks nested-quote +# escaping in some bash versions. +brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh +brev exec "$INSTANCE_NAME" 'export PATH="$HOME/.local/bin:$PATH" && bash ~/reproducer.sh' 2>&1 | tee ./baseline-transcript.log +``` + +**Log-scraping (when `BUG_CLASS=log-only`).** Some bugs describe symptoms that show up in internal log files, not the reproducer's stdout/stderr — e.g., #1642 "see lots of error in openclaw log," #2611 "os.networkInterfaces guard errors." After running the reproducer, also pull the relevant logs from inside the sandbox and search them for the issue's symptom phrase: + +```bash +# Common NemoClaw / OpenClaw / OpenShell log paths inside the sandbox. +brev exec "$INSTANCE_NAME" "sg docker -c 'cat ~/.openclaw/logs/*.log /var/log/nemoclaw/*.log 2>/dev/null'" \ + | tee ./baseline-logs.log + +# Search the log capture for the issue's symptom phrase too, not just the transcript. +grep -F "<symptom phrase from issue body>" ./baseline-logs.log +``` + +For functional bugs the reproducer's stdout is sufficient; for log-only bugs the transcript may be clean but the log capture has the symptom. Both halves feed into the match rubric below. + +**Flake-detection retry.** Even for `functional` bugs, race-prone reproducers (TUI rendering, network policy negotiation, concurrent sandbox state) can produce inconsistent results. Run baseline three times if the first run shows the symptom inconsistently — same script, same env, just three back-to-back invocations. If the three runs disagree, that's signal: + +| 3-run baseline result | Verdict | +|---|---| +| All three reproduce the symptom | Strong baseline match → continue to 8d | +| All three are clean (no symptom) | Reproducer doesn't expose the bug on baseline → Step 8c synth-repro | +| Mixed (1 or 2 of 3 show the symptom) | Flake-prone reproducer. Note "flake suspected" in the comment; apply −25 to Step 9 score; downgrade `+50 latest clean` to `+25` because a clean latest run could just be the lucky path of an intermittent bug | + +Skip flake retry for `performance` and `rebuild-cycle` classes — those have their own multi-run rubrics in Steps 8e and 8f. + +**Match rubric.** LLM compares `baseline-transcript.log` to the issue's "Actual result" / error description. Match criteria, in order: + +1. **Exit code agrees** with what the issue describes (non-zero if issue describes a failure, zero if issue describes a wrong-output bug). Necessary but not sufficient. +2. **Symptom phrase match:** transcript contains a key error phrase from the issue (e.g., issue says `Permission denied on generate-openclaw-config.py`, transcript says `EACCES: permission denied, open '...generate-openclaw-config.py'` — semantic equivalence counts). +3. **Distinguish bug from infra noise:** generic network / DNS / auth errors don't count as a match unless the issue itself describes them. A bug about config parsing that fails at "could not resolve nvidia.com" is an infra failure, not a reproduction. + +**Fallback for issues without an explicit "Actual result" section.** Many bug reports describe a *behavioral* problem rather than a runtime error — e.g., "should default to a stable released version" (#1242), "configuration is not persisted across rebuilds" (#3030). These have no comparable error string. In that case: + +1. Use the issue's **full title + description** as the symptom signal. +2. Match if the reproducer's outcome **contradicts the issue's stated expected behavior** (or matches the stated wrong behavior). E.g., issue says "expected: stable release; actual: nightly", reproducer prints `nightly-build-2026.04.x` → that's a match. +3. If neither error string nor expected-behavior contradiction can be identified, route the script to Step 8c (synth-repro) — let the LLM produce a more diagnostic script that emits something testable. + +- **Match** → reproducer validated. Proceed to 8d. +- **No match** (silent pass, wrong error, infra noise, or no testable outcome): script has gaps. Proceed to 8c. + +### Step 8c: Synth-repro and retry on baseline + +LLM rewrites `./reproducer.sh` using the full issue context (description, environment, symptoms) **plus the baseline transcript** so it can react to what actually happened. Apply **−30 confidence penalty** (or keep it if 8b already applied it for the missing-verbatim case). + +```bash +brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh +brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcript-2.log +``` + +- **Match:** validated (with −30 baked in). Proceed to 8d. +- **Still no match:** mark `verify-inconclusive`. Post a comment that includes both reproducer attempts and both baseline transcripts with the message "couldn't establish a working reproducer for this bug on `$REPORTED_VERSION`." **Skip 8d** — there's nothing to verify on latest. + +### Step 8d: Install latest, run validated reproducer + +```bash +brev exec "$INSTANCE_NAME" "$RESET" +brev exec "$INSTANCE_NAME" " + if [ -f ~/.nvidia-api-key ]; then export NVIDIA_API_KEY=\$(cat ~/.nvidia-api-key); fi + curl -fsSL $INSTALL_URL | bash +" + +# Same resolved-version check as Step 8a — guard against env-var scoping or default fallthrough +# silently installing the wrong version. The latest install should resolve to $LATEST. +RESOLVED=$(brev exec "$INSTANCE_NAME" "bash -lc 'nemoclaw --version'" 2>&1 | tail -1) +echo "[verify-stale] latest requested: $LATEST; resolved: $RESOLVED" +case "$RESOLVED" in + *"$LATEST"*) ;; # match — proceed + *) echo "WARN: latest install resolved to '$RESOLVED' (expected match for $LATEST). Proceeding but flag in comment." ;; +esac + +# OpenShell version pin — surfaced from #1642's e2e run. Latest's blueprint.yaml may set +# `max_openshell_version` below what the OpenShell installer would otherwise grab. The +# baseline phase (Step 8a) installed whichever OpenShell was current at reported-version, +# which can be newer than latest's cap (e.g., reported v0.0.6 → installed openshell 0.0.37, +# latest v0.0.38 caps at 0.0.36, onboard preflight refuses to run). Re-pin from latest's +# repo so onboard preflight passes; if the new pin is OLDER than the installed binary, +# install-openshell.sh refuses the downgrade — fall back to direct GitHub download. +brev exec "$INSTANCE_NAME" ' + set -e + cd ~/NemoClaw + git fetch --depth 1 origin tag "'"$LATEST"'" 2>&1 | tail -2 + git checkout -- . 2>/dev/null || true + git checkout "'"$LATEST"'" 2>&1 | tail -2 + + MAX_OS=$(grep -E "^max_openshell_version:" nemoclaw-blueprint/blueprint.yaml 2>/dev/null | awk "{print \$2}" | tr -d "\"" | tr -d "v") + CUR_OS=$(openshell --version 2>&1 | grep -oE "[0-9]+\.[0-9]+\.[0-9]+" | head -1 || echo 0.0.0) + echo "[verify-stale] openshell pin: blueprint max=$MAX_OS, currently installed=$CUR_OS" + + if [ -n "$MAX_OS" ] && [ "$(printf "%s\n%s\n" "$CUR_OS" "$MAX_OS" | sort -V | tail -1)" != "$MAX_OS" ]; then + echo "[verify-stale] currently installed openshell ($CUR_OS) is newer than blueprint cap ($MAX_OS) — force-downgrading" + sudo rm -f /usr/local/bin/openshell + cd /tmp + curl -fsSL "https://github.com/NVIDIA/OpenShell/releases/download/v$MAX_OS/openshell-x86_64-unknown-linux-musl.tar.gz" -o openshell-pin.tar.gz + tar -xzf openshell-pin.tar.gz + sudo install -m 755 ./openshell /usr/local/bin/openshell + openshell --version + else + sudo bash scripts/install-openshell.sh 2>&1 | tail -3 + fi +' + +brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh +# Same PATH safeguard as the baseline call — non-login shells don't pick up ~/.local/bin +# automatically. The reproducer's internal `sg docker -c '...'` blocks cover Docker access. +brev exec "$INSTANCE_NAME" 'export PATH="$HOME/.local/bin:$PATH" && bash ~/reproducer.sh' 2>&1 | tee ./latest-transcript.log +``` + +If the install of **latest** fails (e.g. installer regression — see #3058 for a current example), this is an infra failure — see Step 11. Do not score or label the issue. + +If install succeeds, `latest-transcript.log` is the input to Step 9 scoring. + +For interactive debugging when something looks off: + +```bash +brev shell "$INSTANCE_NAME" +``` + +--- + +## Step 8d.5: Architectural-Drift Check + +Cross-version verification compares two moving targets: the reproducer assumes `$REPORTED_VERSION`'s tooling surface, and `$LATEST` may have rewritten the surface entirely. If the *tool* the reproducer relies on (CLI subcommand, output table, log file location) was reworked between the two tags, an "empty / clean output on latest" can mean either "bug fixed" OR "we're looking at a deprecated tracking surface." Without this check, the latter silently registers as the former — a class of false positive. + +**Detection** — pickaxe the diff between tags for the reproducer's tool name and watch for the CLI itself being touched, not just its consumers: + +```bash +# Extract the primary verification command from the reproducer (e.g. "openshell forward list"). +TOOL=$(grep -oE '\b(openshell|nemoclaw)[[:space:]]+[a-z-]+' reproducer.sh | sort -u) + +# Pickaxe each tool name across the version range. +for t in $TOOL; do + echo "=== drift check: $t ===" + git log "$REPORTED_VERSION".."$LATEST" -S"$t" --oneline -- src/ bin/ nemoclaw/src/ 2>&1 | head -5 +done +``` + +If a tool is touched, drift is suspected. + +**Multi-axis verification** — when drift is suspected, do not rely on the reproducer's expected output alone. Pick OS-level surfaces that would show the buggy state regardless of which CLI tracks it. For port-forwarding bugs (the #2007 case), the canonical five-axis pattern: + +| # | Surface | Command | +|---|---|---| +| 1 | Reproducer's stated check | as written in the issue body | +| 2 | Host TCP listeners | `sudo ss -tlnp` | +| 3 | iptables NAT redirects | `sudo iptables -t nat -L -n` | +| 4 | Docker port mappings | `docker ps --format '{{.Names}} {{.Ports}}'` | +| 5 | Active SSH tunnels | `ps -ef \| grep 'ssh.*-L'` | + +Adapt the axes to the bug class. For filesystem bugs: `find`, `lsattr`, `stat`. For network policy bugs: `iptables -L`, container netns, gateway logs. The principle is the same — pick at least three independent surfaces that would each independently show the buggy state if it were present. + +**Action when drift is suspected:** + +- Run the multi-axis pattern after Step 8d's reproducer. +- The verdict requires **every relevant axis to be clean** — not just the reproducer's surface — before claiming `fixed-on-latest`. +- Quote the multi-axis evidence in the Step 10 comment as a table; this is exactly what makes "fixed" defensible when the original tooling no longer reflects the underlying behavior. +- If any axis still shows the buggy state, the bug is NOT fixed even if the reproducer's surface is clean. Escalate to "still reproduces" (Step 9 special case). + +**When drift is NOT suspected** (the reproducer's tool is unchanged in the version range): the reproducer's expected output is sufficient, no multi-axis verification needed. + +--- + +## Step 8e: Performance-Bug Verification (when `BUG_CLASS=performance`) + +Performance bugs (#2598 "10s P50", #2600 "hangs ~2 min", #2733 Ollama tool-call leak over time) can't be answered by the standard exit-code + symptom-phrase rubric — one clean reproducer run doesn't tell you the p50 budget is met; one slow run doesn't tell you the bug still reproduces. Replace Step 8b's match with a measurement-and-distribution rubric: + +1. **Parse the SLA from the issue body.** Extract numeric latency thresholds: `10s P50`, `200ms`, `under 5 seconds`, `~2 min`. Save as `SLA_P50_MS`, `SLA_P90_MS`, etc. If no numeric SLA is in the body, route to Step 8c synth-repro to ask the reporter (via comment) for one — without a target, the verdict is undefined. +2. **Run the reproducer N=10 times** on each side (baseline + latest), capturing per-run latency: + + ```bash + for i in $(seq 1 10); do + /usr/bin/time -f '%e' bash ~/reproducer.sh >/dev/null 2>>./latest-perf.log + done + ``` + +3. **Compute p50 and p90** for both sides. `sort -n ./latest-perf.log | awk 'NR==5'` for p50 of 10 runs. +4. **Match rubric:** + - Latest's p50 within the SLA AND baseline's p50 outside the SLA → bug fixed; same Step 9 scoring (subject to baseline-validation gate). + - Latest's p50 outside the SLA → bug still reproduces (Step 9 special case). + - Latest p50 within SLA AND baseline p50 also within SLA → reproducer doesn't actually exercise the bug; route to Step 8c synth-repro. + +**Hardware-substitution caveat.** Performance numbers are silicon-dependent. When the issue is `Platform: DGX Spark` or `Platform: GB10` and we're measuring on a Brev x86 GPU SKU, the comment must say so explicitly: a Brev p50 of 1.5s on a `H100` does not prove the DGX Spark p50 is fixed. Cap the score at 60 unless the bug is clearly silicon-independent (e.g. an algorithmic regression in user-space JS that would manifest the same on any silicon). + +--- + +## Step 8f: Rebuild-Cycle Verification (when `BUG_CLASS=rebuild-cycle`) + +Rebuild-cycle bugs (#2701 "Pod recreate wipes `/tmp/nemoclaw-proxy-env.sh`," issues describing "configuration is not persisted across rebuilds") only manifest when sandbox state crosses a destroy/recreate boundary. A single onboard run can't trigger the symptom. Replace Step 8b's match with a run-rebuild-rerun harness: + +1. **First onboard.** Run the reproducer once to establish initial state. Capture relevant artifacts (config files, env vars, sandbox metadata) — the issue body usually names what should persist: + + ```bash + brev exec "$INSTANCE_NAME" "sg docker -c 'cat <files-mentioned-in-issue> 2>&1'" | tee ./pre-rebuild.log + ``` + +2. **Trigger the rebuild.** Use `nemoclaw destroy --all --force` followed by `nemoclaw onboard` with the same env vars. Do NOT comprehensive-reset between (the point is to test the destroy/recreate, not start from scratch). + +3. **Re-capture the same artifacts** post-rebuild: + + ```bash + brev exec "$INSTANCE_NAME" "sg docker -c 'cat <same-files> 2>&1'" | tee ./post-rebuild.log + ``` + +4. **Diff and match.** The bug is "X gets wiped / changes / regresses across rebuild." Compare pre-rebuild vs post-rebuild captures to the issue's expected behavior: + - Pre and post agree (artifact preserved) AND issue says it should be preserved → bug fixed + - Pre and post differ (artifact wiped) AND issue says it gets wiped → bug still reproduces + - Pre and post agree AND issue says it gets wiped → reproducer doesn't exercise the bug; Step 8c synth-repro + +The harness still uses Step 9's scoring framework — `+50 latest clean (artifact preserved)`, etc. — but the "what gets compared" axis is the diff, not the symptom phrase. + +--- diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/scoring-comments-and-logging.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/scoring-comments-and-logging.md new file mode 100644 index 00000000000..13ba191d843 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/scoring-comments-and-logging.md @@ -0,0 +1,496 @@ +<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> +<!-- SPDX-License-Identifier: Apache-2.0 --> + +# verify-stale — Scoring, Comments, Labels, and Logging Reference + +Use after a latest result exists or after a by-design/inconclusive branch is selected. Covers confidence scoring, redaction, concise comments, labels, project movement, infra failures, and activity logging. + +## Contents + +- [Step 9: Score Confidence](#step-9-score-confidence) +- [Step 10: Compose and Post the Comment](#step-10-compose-and-post-the-comment) +- [Step 11: Infra Failure Handling](#step-11-infra-failure-handling) +- [Step 12: Log to Activity](#step-12-log-to-activity) +- [Cadence](#cadence) +- [Out of Scope (v1)](#out-of-scope-v1) +- [Companion Behavior](#companion-behavior) + +--- + +## Step 9: Score Confidence + +Start at 0. Apply each rule that fires. + +| Signal | Delta | +|---|---| +| Reproducer ran cleanly on **latest** (8d), exit 0, no bug symptom observed | +50 | +| Commits between reported version and `$LATEST` touch the implicated component (see "Path extraction" below) | +25 | +| A merged PR mentions this issue number or its symptom (see "PR search" below) | +25 | +| Reproducer was LLM-synthesized at any point (Step 8b synth or Step 8c retry) | −30 | +| Any partial error, warning, or flaky behavior in the latest run (8d) | −50 | + +Total is clamped to `[0, 100]`. + +### Path extraction (for the +25 commits signal) + +The skill needs to know *which* path to `git log v<reported>..$LATEST -- <path>` against. Apply in order, stop at the first that yields a non-empty path: + +1. **Stack trace / file path mentions in the issue body.** Grep the body for absolute paths under known install roots, then map to repo paths: + - `/usr/local/lib/nemoclaw/<rel>` → `<rel>` in repo (e.g., `scripts/generate-openclaw-config.py`) + - `/usr/local/bin/nemoclaw*` → `bin/` + - `~/.nemoclaw/<rel>` → most often runtime state, drop unless the bug is config-related → `src/lib/config/` + - In-repo paths (e.g., `bin/lib/policies.js` mentioned literally) → use as-is +2. **Component-label-to-directory map.** Pick the first match. Paths verified against the current repo layout — drop any path that doesn't exist on the tag at `$LATEST` rather than passing it to `git log`. + - `NemoClaw CLI` → `bin/`, `src/lib/`, `nemoclaw/src/commands/` + - `Sandbox` → `nemoclaw/src/blueprint/`, `nemoclaw-blueprint/` + - `OpenShell` → cross-repo (lives at `github.com/NVIDIA/OpenShell`, not in this repo). Skip the +25 signal for OpenShell-only issues; cross-repo `git log` is out of v1 scope. + - `Docker` → `Dockerfile`, `Dockerfile.base`, `scripts/install-openshell.sh`, `scripts/install.sh` + - `Getting Started` → `docs/`, `scripts/install.sh` + - `Integration: <X>` — no `src/lib/integrations/` exists in this repo. Skip the +25 signal for integration-component issues unless source 1 (file paths in body) yielded a path. +3. **Title keywords.** "policy" → `nemoclaw-blueprint/policies/`, `nemoclaw/src/blueprint/`. "inference" → `docs/inference/` is docs-only; skip the +25 signal unless source 1 surfaces actual code paths. + +If none of the above produces a path, **skip the +25 signal entirely** rather than guessing. Floating the +25 on every issue would inflate scores meaninglessly. + +### PR search (for the +25 PR signal) + +```bash +# Direct issue-number reference (covers most cases — "fixes #2861" etc.) +DIRECT_REF=$(gh pr list --repo NVIDIA/NemoClaw --state merged \ + --search "$ISSUE_NUMBER" \ + --json number,title,mergedAt,body \ + -q "[.[] | select((.body + \" \" + .title) | test(\"#$ISSUE_NUMBER\\\\b\"))]") + +# Symptom-phrase fallback (only if direct reference returns nothing) +if [ -z "$DIRECT_REF" ] || [ "$DIRECT_REF" = "[]" ]; then + SYMPTOM=$(extract first key error/symptom phrase from issue body, ~3-6 words) + SYMPTOM_REF=$(gh pr list --repo NVIDIA/NemoClaw --state merged \ + --search "\"$SYMPTOM\"" \ + --json number,title,mergedAt) +fi +``` + +Apply +25 if either query returns at least one PR with `mergedAt` strictly after the tag date of `$REPORTED_VERSION` (look up via `git log -1 --format=%cI v$REPORTED_VERSION`). PRs merged before the reporter even filed the issue can't have fixed it. + +If neither query returns anything, **skip the +25 signal**. + +**Baseline-validation gating.** The +50 weight assumes the reproducer was *validated* — i.e., it produced the bug symptom on baseline (Step 8b/8c match). If `BASELINE_INSTALL_FAILED=1` (Step 8a fall-through, baseline pass skipped — including the sandbox-build-rot case from Step 11), the +50 still applies but **cap the total at 84**. Corroboration signals (commits-touched-area, PR-mention) still raise the score within the cap but cannot lift it above 84. Without runtime baseline confirmation we don't have enough on our own to claim ≥85 — the cap forces the verdict into the 60–84 band where the reporter is asked to confirm. The previous draft of this rule had an "unless commits-touched OR PR-mention also fires" escape hatch that let inferred fix evidence bypass the cap entirely; that produced a misleading 100/100 on the #2007 e2e run despite zero baseline confirmation, and was tightened here. + +**Action (when latest run was clean — bug not reproduced):** + +| Score | Label | Comment | +|---|---|---| +| ≥85 | `fixed-on-latest` | Evidence-rich, no @-mention. | +| 60–84 | `fixed-on-latest` | Evidence-rich, **@-mention the original reporter** to confirm. | +| <60 | `verify-inconclusive` | Short, honest "couldn't verify" explanation. | + +**Special case: latest output matches the issue symptom (bug still reproduces on latest).** + +This is not a flake — the skill positively confirmed the bug is still live. Don't apply the +50 weight (the bug isn't fixed) and skip the score table entirely. + +- Post a "still reproduces on latest" comment with both transcripts. +- Apply **no label**. +- Include the marker `<!-- nemoclaw-verify-stale v1 YYYY-MM-DD -->` with today's date so the candidate filter applies the 7-day TTL (Step 3 idempotency). +- Next weekly run picks the issue back up after the TTL — if the bug gets fixed in the meantime, that run catches it. + +The skill **never closes issues** in any branch. A maintainer pulls that trigger after reviewing the label and comment. + +--- + +## Step 10: Compose and Post the Comment + +**Redaction pass before posting.** Run on **every** chunk of text quoted in the comment — issue body excerpts, baseline transcript, latest transcript, synth-repro scripts. Replace each match with `[REDACTED]`. The transcripts especially leak — they include full stdout/stderr from real installs and runs. + +**HTML → text pre-pass for issue body excerpts.** NV QA bodies are HTML; tokens nested in `<pre>` tags or HTML attributes (e.g. `<a href="https://user:tok@host/...">`) slip past the regex patterns below if the input still has tags. Convert to plain text first, then redact: + +```bash +TEXT=$(printf '%s' "$BODY_EXCERPT" | python3 -c ' +import html, re, sys +b = sys.stdin.read() +b = re.sub(r"<br\s*/?>", "\n", b) +b = re.sub(r"</?(p|div|tr|td|th|li|pre)[^>]*>", "\n", b) +b = re.sub(r"<[^>]+>", "", b) +print(html.unescape(b)) +') +# Now apply the regex table below to $TEXT. +``` + +Transcripts and synth-repro scripts are already plain text and skip the pre-pass. + +**Order matters and the patterns below are in execution order.** Longest, most-specific patterns first; generic catchalls last. Otherwise the catchall masks specific matches and you lose track of what was actually redacted (JWT vs session blob vs random base64). + +Patterns live in a fenced block (not a markdown table) because patterns 8 and 9 use regex alternation `|` — markdown tables would treat the literal `|` as a column delimiter, and escaping it as `\|` makes the regex match a literal pipe instead of an alternation, which silently breaks credential redaction. + +```regex +1. eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,} + → JWT tokens + +2. gh[pousr]_[A-Za-z0-9]{36,} + → GitHub PATs / install tokens + +3. (?i)nvapi-[A-Za-z0-9_-]{20,} + → NVIDIA API keys (NIM / build.nvidia.com) + +4. AKIA[0-9A-Z]{16} + → AWS access key IDs + +5. (?i)aws_secret_access_key\s*=\s*\S+ + → AWS secret keys + +6. (?i)authorization:\s*\S+ + → HTTP auth headers (often Bearer + JWT) + +7. URLs containing `@` before the host (e.g., https://user:pw@host/...) + → Basic-auth credentials in URLs + +8. (?i)(token|secret|password|api[_-]?key|bearer)[^\n]*[:=][^\n]* + → Inline credentials in env/config/log output + +9. \b\w+\.(nvidia\.internal|nv-internal\.com|nvidia\.dev)\b + → Internal hostnames (extend list per team) + +10. [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} + → Email addresses (PII) + +11. \b[A-Za-z0-9+/]{60,}={0,2}\b + → Long base64 blobs (likely keys/sessions; tune length to taste — too short hits legit data) +``` + +**File paths under the reporter's home directory** (`/Users/<name>/`, `/home/<name>/`) → replace with `~/`. Run last; catches incidental username PII. + +**Comment authoring principle.** Every section in a rendered comment must either change a reader's mind about the verdict, or be cut. Word counts follow from that — **300 is a hard ceiling** for the main verdicts (fixed-on-latest, wontfix). Simple cases (clear PR ref, deterministic check) land under 200. The principle generalizes: comments posted by this skill compete for a maintainer's attention against every other in-flight thread, and "AI-slop" prose — architectural sidebars, file:line citations the maintainer can find via the PR ref, bare-output reproductions when the load-bearing evidence is elsewhere, "if this verification is wrong, please reopen…" boilerplate — actively reduces the comment's signal-to-noise ratio. + +**For each section in a draft, ask: would the maintainer reach a different conclusion *without* this section? If no, delete.** Lessons accumulated from real runs: + +- **#2007 first draft (~750 words):** had a multi-paragraph "Architectural notes for QA reference" section that didn't change the verdict. Cut → 371 words. +- **#2604 first three drafts:** wavered between fixed-on-latest, still-reproduces, and by-design across iterations because each draft padded the verdict with prose that didn't ground it. Final 190-word draft cut a maintainer-note sidebar about platform attribution, a bare-status output reproduction, and a file:line citation of the source — none affected the verdict, all were AI-slop padding. Rule learned: **before drafting any prose, name the verdict in one sentence; if a section doesn't directly support that one sentence, cut it before writing it.** + +**Per-verdict length defaults:** + +| Verdict | Target | Rationale | +|---|---|---| +| `fixed-on-latest` | **200–300 words** | Header + evidence + verdict + @-mention. Add hardware-substitution caveat or related-failure-mode section only if they shift the maintainer's read. If you're past 300, you're padding. | +| `wontfix` (by-design) | **200–300 words** | Structurally-fixed + vestigial + what's-not-the-same-bug, each one to two sentences max. The PR ref carries the detail; the comment carries the verdict. | +| `verify-inconclusive` | 100–200 words | One paragraph naming what the skill couldn't establish. No transcripts beyond a single quoted line. | +| **Still-reproduces (no label)** | **30–80 words** | The reporter already has the symptom; the maintainer can see the issue is open. The skill is just confirming + setting the TTL marker. **No transcripts** (the issue body has them), **no @-mention** (the reporter knows their bug is real), **no architectural prose**. One sentence stating "skill ran reproducer on `<latest>`, symptom still present" + one sentence on any partial-fix PR if relevant + marker. That's it. | + +**Cut, by default:** + +- Maintainer-note sidebars about labels / platform attribution unrelated to the bug surface. +- Bare-output reproductions when the load-bearing evidence is in a different command's output. +- File:line citations of source code already findable via the cited PR. +- Closing "if this verification is wrong, please reopen…" boilerplate. +- Redundant verbal framing of what the evidence already shows ("the table above proves…"). +- "Verification mode" pleasantries beyond one factual line. + +**Mandatory cap caveat.** When the score is capped (Step 9 baseline-validation gating, or any Step 11 degraded-mode path), the rendered Verdict section must include a one-line caveat naming the cap and the reason. Example: `Capped at 84 because Step 9's baseline-validation gate did not run (sandbox-build rot on v0.0.18: Dockerfile symlink layer removed by #2227).` Don't make readers reverse-engineer why the score didn't go higher — name it. + +**Mandatory hardware-substitution caveat.** When the issue carries `Platform: DGX Spark` or `Platform: GB10` and Step 7 provisioned a Brev SKU that is not the same silicon (Brev's stoppable GPU catalog is x86 + discrete H100/A100/L40S/T4 — not Grace Hopper / GB10 unified-memory ARM64), the rendered comment must include a one-line "Hardware substitution" note. Example: `Hardware substitution: verified on Brev n1-standard-4:nvidia-tesla-t4 (x86_64 + T4) as a substitute for the reporter's DGX Spark (ARM64 + GB10). For silicon-shape bugs (perf, memory architecture, drivers) this is not a faithful repro — please confirm on actual DGX Spark.` This goes in the metadata block right after `Verification mode:` so it's visible at the top, not buried in the analysis. + +**Mandatory `Verification mode` header line.** All three templates below include a `**Verification mode:**` line in the metadata block, naming what we did and didn't actually run (e.g., "runtime reproduction on Brev <SKU>; baseline + latest both installed and run" for the standard template; "static analysis at the verified-on tag — no runtime reproduction" for the by-design template; "runtime reproduction on Brev <SKU>; bug confirmed live on latest" for still-reproduces). Reader should never have to guess whether the verdict came from real install logs or from static analysis. + +**Link-pass self-verification (all templates).** Same rule as Step 8.5d's link pass, applied to every template. Resolve at least one rendered markdown link from each section that has them (`What's structurally fixed` / `Vestigial references` / `Existing CI coverage` for by-design; `Relevant changes since` / transcript code-anchor citations for the standard template) via `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 if path exists at tag, 404 otherwise) or `curl -fsI <blob-url>`. A 404 on a citation in the rendered comment is worse than no citation — it advertises verification work that didn't actually happen. If any link fails to resolve, fix it or bail to `verify-inconclusive`. + +**Mandatory closing block — reporter @-mention with confirmation language.** Every template below **except `Still-reproduces`** ends with an explicit @-mention of the original reporter using this exact shape: + +> @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. + +The skill cannot independently confirm a closed-as-fixed verdict — only the reporter knows whether their original symptom is gone in their environment. The @-mention is what converts a "skill says it's fixed" claim into actionable confirmation work for QA. Customize `<Z>` per case (the version that shipped the fix or `$LATEST`), but never omit the line. + +**Mandatory unanswered-question prefix and dual @-mention.** When Step 3 sets `UNANSWERED_MAINT_LOGIN` (a maintainer's question is older than 7 days and the reporter never replied), the verdict comment changes shape in two places: + +1. **Prepend a lead paragraph** as the very first line of the body, before the `## Stale-issue verification` heading. The lead paragraph is a single line: + + ```text + [@UNANSWERED_MAINT_LOGIN's comment](UNANSWERED_MAINT_URL) from UNANSWERED_MAINT_DATE is still unanswered. Posting independent verification below to unstick the thread. + ``` + + …with the bracketed variables expanded from the values exported by Step 3. + +2. **Replace the closing reporter-only @-mention with a dual @-mention** that names BOTH the maintainer (acknowledging the open question) and the reporter (per the standard confirmation pattern): + + > @\<UNANSWERED_MAINT_LOGIN\> — flagging that your question above is still open; the verification below may answer it. @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. + +This applies to all three templates (fixed, still-reproduces, by-design). The skill becomes the *unsticking voice* on a thread that has gone quiet — never a clueless interruption when discussion is fresh (Step 3 already filtered the within-7-day case). + +**Comment template (fixed / inconclusive — bug not reproduced on latest):** + +````markdown +## Stale-issue verification — automated + +**Reported on:** v0.0.31 +**Verified on:** v0.0.34 (commit abc1234) +**Verification mode:** runtime reproduction on Brev `<instance-class>` — baseline (v0.0.31) and latest (v0.0.34) both installed and run; comparison made on the captured transcripts. (Or: "runtime reproduction on Brev `<instance-class>` — baseline-install-skipped (`.openclaw-data` rot, see Step 11), latest-only run; verdict capped at 84.") +**Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 / <CUDA version if GPU> + +### Baseline (reported version) + +- Install: succeeded · skipped (install rotted) +- Reproducer: extracted verbatim · synthesized (−30 penalty) +- Result: bug symptom matched (validated) · could not validate (skipped Step 8c gate) + +<details><summary>Baseline transcript</summary> + +```text +<full baseline transcript> +``` + +</details> + +### Latest + +- Install: succeeded +- Result: not reproducible — clean run, no bug symptom observed + +<details><summary>Latest transcript</summary> + +```text +<full latest transcript> +``` + +</details> + +### Verdict + +**Confidence:** 88 / 100. Labelling `fixed-on-latest`. + +<details><summary>Relevant changes since v0.0.31</summary> + +- abc1234 — fix: <commit subject> +- def5678 — refactor: <commit subject> + +</details> + +@<reporter> — please confirm the symptom is gone on a recent build (≥ v0.0.<Z>) and reopen with a fresh reproducer if you observe otherwise. + +<!-- nemoclaw-verify-stale v1 2026-05-12 --> +```` + +**Comment template (still reproduces — Step 9 special case):** + +````markdown +## Stale-issue verification — still reproducible + +**Reported on:** v0.0.31 +**Verified on:** v0.0.34 (commit abc1234) +**Verification mode:** runtime reproduction on Brev `<instance-class>` — baseline confirmed the symptom matches the issue; latest (v0.0.34) also produced the symptom. Bug is still live. +**Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 + +The skill ran the reported reproducer on v0.0.34 and observed the same bug symptom described in this issue. The bug is still live. + +No label applied. Will re-verify automatically next weekly run; if a fix lands in the interim, the next pass catches it. + +@<reporter> — please confirm the symptom still matches your observation on v0.0.<Y> and reopen with any updated reproducer or environment details if it has shifted. + +<details><summary>Baseline transcript (validated reproducer)</summary> + +```text +<baseline transcript> +``` + +</details> + +<details><summary>Latest transcript (bug still observed)</summary> + +```text +<latest transcript> +``` + +</details> + +<!-- nemoclaw-verify-stale v1 2026-05-12 --> +```` + +The trailing HTML comment is the **idempotency marker** Step 3 looks for. Always include today's date in `YYYY-MM-DD` format so the candidate filter can apply the 7-day TTL. + +**Pre-post state-check.** A long-running verification can race with the maintainer closing the issue independently — happened on #2513 and #2519 (mid-batch closes by @jyaunches with their own verification). Re-check `state == OPEN` right before posting. If closed, apply the label tag-only (skipping the comment, since the maintainer's own close-comment is now the authoritative record) and skip the Project 199 move. + +```bash +STATE=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json state --jq .state) +if [ "$STATE" != "OPEN" ]; then + echo "[verify-stale] #$ISSUE_NUMBER closed since verification started — applying label tag-only, skipping comment + tracker move" + gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "$LABEL" + exit 0 +fi +``` + +**Post the comment and apply the label:** + +```bash +gh issue comment "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --body-file comment.md +gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "fixed-on-latest" +# or for <60: +# gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-label "verify-inconclusive" +``` + +**Move the issue to "Needs Review" on the NemoClaw Development Tracker AND self-assign (only on `fixed-on-latest`).** The tracker is GitHub Project [NVIDIA/199](https://github.com/orgs/NVIDIA/projects/199) ("NemoClaw Development Tracker"). When the skill's verdict is `fixed-on-latest`, the issue moves to **Needs Review** AND the issue is assigned to the maintainer who ran the skill (`$GH_IDENTITY` from Step 6.5) — assignment puts the issue in their personal review queue so they don't lose track of what they've staked their name on. After the reporter confirms and the maintainer closes, existing Project automation (or a manual move) advances it to Done. **No move and no assign on `wontfix` / `verify-inconclusive` / no-label-still-reproduces** — those have separate close paths. + +This step requires the `project` scope on the maintainer's gh CLI (`gh auth refresh -h github.com -s project` in a real terminal once; OAuth device-code flow). If the scope is missing, the lookup query returns an auth error — fall through with a one-line warning rather than failing the whole run. + +```bash +# Project 199 constants (re-run gh project field-list 199 --owner NVIDIA --format json +# if the project gets renamed/restructured and these IDs drift): +PROJECT_ID="PVT_kwDOABpemM4BSCP5" +STATUS_FIELD_ID="PVTSSF_lADOABpemM4BSCP5zg_r9p8" +NEEDS_REVIEW_OPTION_ID="5c5922a9" + +# Only fire on fixed-on-latest. Skip silently otherwise. +if [ "$VERDICT" = "fixed-on-latest" ]; then + # Find the issue's existing project item, if any. + ITEM_ID=$(gh api graphql -f query=' + query($num: Int!) { + repository(owner: "NVIDIA", name: "NemoClaw") { + issue(number: $num) { + projectItems(first: 10) { + nodes { id project { number } } + } + } + } + }' -F num="$ISSUE_NUMBER" \ + --jq '.data.repository.issue.projectItems.nodes[] | select(.project.number == 199) | .id' \ + 2>/dev/null | head -1) + + # If the issue isn't on the project yet, add it. (NV QA bots usually add new + # issues automatically, but cover the gap.) + if [ -z "$ITEM_ID" ]; then + ITEM_ID=$(gh project item-add 199 --owner NVIDIA \ + --url "https://github.com/NVIDIA/NemoClaw/issues/$ISSUE_NUMBER" \ + --format json --jq .id 2>/dev/null) + fi + + if [ -n "$ITEM_ID" ]; then + gh project item-edit \ + --id "$ITEM_ID" \ + --project-id "$PROJECT_ID" \ + --field-id "$STATUS_FIELD_ID" \ + --single-select-option-id "$NEEDS_REVIEW_OPTION_ID" \ + >/dev/null && echo "[verify-stale] moved #$ISSUE_NUMBER to 'Needs Review' on Project 199" + else + echo "[verify-stale] WARN could not resolve project item for #$ISSUE_NUMBER on Project 199 — label applied but tracker not moved" + fi + + # Self-assign the issue to the maintainer who ran the skill — puts it in their + # personal review queue alongside the Needs Review state. + gh issue edit "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --add-assignee "$GH_IDENTITY" \ + >/dev/null && echo "[verify-stale] assigned #$ISSUE_NUMBER to @$GH_IDENTITY" +fi +``` + +The Step 12 activity log line should record the project move (or the warn-and-skip case) so a maintainer scanning the log can spot tracker drift. Add a `Tracker:` row to the per-issue entry: `Tracker: moved to Needs Review` | `not moved (verdict: <X>)` | `not moved (project lookup failed)`. + +--- + +## Step 11: Infra Failure Handling + +Two different failure types, two different responses. + +**Latest-install failure** (Step 8d) or reuse-check / provisioning / harness errors: hard infra failure. + +- Print the error. +- Apply **no label** — infra failures must not pollute the verification record. +- Post a short comment **only if explicitly requested by the invoking user**. Default is silent move-on. +- Continue to the next candidate in batch mode. + +The next weekly run retries naturally. + +**Baseline-install failure** (Step 8a, reported version won't install on a modern image): not a hard failure — degraded mode. + +- Set `BASELINE_INSTALL_FAILED=1`, skip 8b/8c, jump to 8d. +- Step 9 applies the score cap (max 84) — corroboration signals raise the score within the cap but cannot lift past it. +- Note "baseline-install-skipped" in the final comment so a reviewer knows the verification ran without the script-validation gate. + +**Baseline-build failure** (Step 8a binary install succeeded, but the in-image `Dockerfile` build during sandbox creation failed on a layer that was structurally removed in a later release): also degraded mode, distinct from binary install rot. Surfaced during the #2007 e2e run on v0.0.18 (`/sandbox/.openclaw-data/workspace/media` symlink layer, removed entirely by #2227). + +- Set `BASELINE_INSTALL_FAILED=1` (same flag — Step 9's cap-at-84 rule keys off it regardless of which phase rotted). +- Skip 8b/8c, jump to 8d. +- Note "baseline-build-skipped" in the final comment with the specific failing layer/file so a reviewer can see *why* the v0.0.X image no longer builds (the why is usually a follow-on PR that removed the rotted layer). +- Do not retry the build with a patched Dockerfile — that breaks faithfulness. We're claiming "couldn't independently re-trigger the original symptom on baseline," not "we made the old version work somehow." + +Both baseline-rot variants share the same downstream effect: Step 9 cap, Step 10 caveat, @-mention reporter to confirm. Distinguishing them in the comment helps a reviewer understand the failure mode without re-running. + +This degradation is expected — old releases rot at multiple phases (binary installer URL drift, base-image dependencies vanish, in-image Dockerfile layers get removed by structural refactors). We still want to extract whatever signal we can from the latest run plus PR/commit evidence, just at a more conservative confidence ceiling. + +**Empirical reality after two e2e runs:** baseline-build-rot is the **dominant** failure mode for any reported version more than ~5–7 patches behind, not an edge case. Both #2007 (v0.0.18, 17 patches behind) and #2592 (v0.0.28, 7 patches behind) hit it. The cap-at-84 with reporter @-mention is the **modal** verdict shape for stale-issue verification, not the exception. Reframe expectations accordingly: + +- For issues reported >5 patches behind `$LATEST`, plan for the cap-at-84 path. Pre-flight (PR-search, pickaxe) carries more weight than baseline runtime evidence. +- For issues reported within 1–4 patches of `$LATEST`, baseline is more likely to install cleanly and the full +50 path is reachable. +- The skill's design assumes baseline + latest both run cleanly; in practice latest-only with cap-at-84 is the workhorse path. The score-cap is doing real work, not just a fallback. + +**Keep-box-on-inconclusive.** When `verify-inconclusive` lands (Step 8c gave up, or Step 9 score < 60), **skip the cleanup trap** for this run if the box was provisioned by this run — set `PROVISIONED_NEW=0` before the trap fires so the EXIT handler is a no-op. Print the `brev shell "$INSTANCE_NAME"` command and an explicit `brev delete "$INSTANCE_NAME"` reminder in the run output so the maintainer can triage and clean up manually. Reused boxes stay regardless. Ship-failed verifications are the exact case where having an inspectable artifact pays for itself; an unbounded sleep-and-delete in the background isn't reliable across session ends, so we leave deletion explicit. + +--- + +## Step 12: Log to Activity + +After each issue (verified, inconclusive, by-design, or infra-failed), append to `${VERIFY_STALE_LOG_DIR:-$HOME/development/daily-rhythm/activity}/nemoclaw-verify-stale-log.md`. The default path matches the personal-organizer convention; export `VERIFY_STALE_LOG_DIR` to point elsewhere (CI, shared volume, etc.). Create the directory if missing — do not assume it exists. + +```markdown +### NVIDIA/NemoClaw#<number> — <title> +**Date:** YYYY-MM-DD +**Reported on:** v0.0.31 +**Verified on:** v0.0.34 +**Environment:** CPU | GPU (<instance type>) +**Box:** reused <name> | provisioned <name> | local (no Brev — Step 6.7 short-circuit) +**Baseline install:** succeeded | failed (degraded mode) +**Baseline match:** validated (verbatim) | validated (synth) | failed (verify-inconclusive) | skipped +**Latest install:** succeeded | failed (infra error) +**Latest result:** not-reproduced (clean) | still-reproduces | partial / flake | n/a (skipped 8d) +**Confidence:** 88 / 100 | n/a (still-reproduces) +**Label applied:** fixed-on-latest | verify-inconclusive | status: wont-fix | none (still-reproduces) | none (infra) +**Tracker:** moved to Needs Review on Project 199 | not moved (verdict: <X>) | not moved (project lookup failed) +**Assignee:** @<GH_IDENTITY> | not assigned (verdict: <X>) +**Brev wall time (approx):** N min + +--- +``` + +Create the file if missing, with this header: + +```markdown +# NemoClaw — Verify Stale Log + +A running record of stale-issue verification runs on NVIDIA/NemoClaw. +Persisted via daily-rhythm to GitLab. + +--- +``` + +At end of a batch session, prepend a session summary: + +```markdown +## YYYY-MM-DD — Verify Session +**Issues considered:** N +**Verified `fixed-on-latest`:** N +**Marked `status: wont-fix` (by-design path):** N +**Marked `verify-inconclusive`:** N +**Local-first short-circuits (no Brev cost):** N +**Skipped (Windows / macOS / integration / no version):** N +**Infra failures:** N +**Brev wall time:** N min · approx $X.XX + +--- +``` + +Never stage or commit the log to the NemoClaw repo. + +--- + +## Cadence + +- **Weekly cron** — Monday morning, batch mode, ≤15 issues (the Step 1 cap, sliced after Step 3/4 filters). +- **Manual** — invoke with a single issue number anytime. + +--- + +## Out of Scope (v1) + +- Auto-closing issues. Always tag-only; a human pulls the trigger. +- macOS verification *via the Brev path*. Brev offers no macOS instances. The Step 6.7 local-first short-circuit *does* run on a maintainer's macOS laptop — so manual single-issue runs against pure-CLI bugs work on macOS. The weekly batch cron is Linux-only because that path always uses Brev. +- Issues requiring third-party integration credentials (Slack, Discord, Telegram, Hermes, OpenClaw, WeChat). +- Service-account bot identity. v1 runs under each maintainer's own GitHub credentials. +- Versioned labels. A single `fixed-on-latest` label is swept on each release cut. + +--- + +## Companion Behavior + +`nemoclaw-maintainer-cut-release-tag` sweeps `fixed-on-latest` and `verify-inconclusive` from all open issues at release time. Without that sweep, "latest" drifts and verifications go stale silently. The by-design path uses the existing repo `status: wont-fix` label; that label is **not** swept (it's also applied for non-skill reasons such as scope or priority decisions, and clearing it would erase human triage work). From 33a350c59d0d443142556c6458252e3c9ea86d4a Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Tue, 26 May 2026 09:17:30 -0700 Subject: [PATCH 38/40] fix(verify-stale): address 11 CodeRabbit findings on PR #3327 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - brev-provisioning: append `// empty` to the CPU_TYPE jq filter so an empty result yields an empty string instead of the literal "null", which `[ -n "$CPU_TYPE" ]` would otherwise accept and pass to `brev create --type "null"`. - by-design: fix "greping" -> "grepping". - candidate-selection: drop the hardcoded "8 prefixed variants" count (the inline list enumerates 11) and point readers at `gh label list --search enhancement` for the live set. - environment-and-reproducer (API key entry): replace the inline `printf '%s' '<your-key>' > ~/.nvidia-api-key` example with a no-echo, no-history `IFS= read -rs` flow; the inline form left the key in shell history. - environment-and-reproducer (local-first match): split the conjoined predicate into two explicit branches — matches reported symptom -> still-reproduces, matches expected-fixed behavior -> fixed-on-latest; third branch falls through to Brev as before. - reproduction-rubrics (Step 8c rerun): prepend the same `export PATH="$HOME/.local/bin:$PATH"` guard 8b and 8d use, so a synth-repro rerun doesn't misread as `verify-inconclusive` because the nemoclaw binary fell off PATH. - reproduction-rubrics (drift-check loop): switch from `for t in $TOOL` to `mapfile -t TOOLS` + `for t in "${TOOLS[@]}"`, so multi-word tool strings ("openshell forward") stay intact under pickaxe instead of word-splitting into separate searches. - reproduction-rubrics (Step 8e perf rubric): compute p50 as the mean of the 5th and 6th sorted values for N=10 (standard median), and add the missing p90 line as the nearest-rank 9th value. Match rubric grows a p90 backstop that flips a within-p50 verdict to still-reproduces when an issue-declared p90 SLA is missed. - scoring-comments-and-logging (idempotency marker): replace the two hardcoded `2026-05-12` template dates with the `YYYY-MM-DD` placeholder the surrounding prose already calls for. - scoring-comments-and-logging (Markdown capitalization): capitalize "Markdown" in two prose mentions. - scoring-comments-and-logging (still-reproduces contract): make the per-verdict table on L174 canonical — strip the closing reporter @-mention, baseline transcript, and latest transcript from the template body so it actually fits the 30–80-word target. Scope the unanswered-question dual @-mention rule to fixed-on-latest and by-design only (still-reproduces has no closing @-mention to replace); the lead-paragraph half of the rule still applies to all three templates. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../reference/brev-provisioning.md | 2 +- .../reference/by-design.md | 2 +- .../reference/candidate-selection.md | 2 +- .../reference/environment-and-reproducer.md | 21 +++++--- .../reference/reproduction-rubrics.md | 33 ++++++++++--- .../reference/scoring-comments-and-logging.md | 49 ++++++------------- 6 files changed, 58 insertions(+), 51 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/brev-provisioning.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/brev-provisioning.md index b6ebe106715..c30a0af4e56 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/brev-provisioning.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/brev-provisioning.md @@ -75,7 +75,7 @@ else CPU_RAM_FLOOR=${CPU_RAM_FLOOR:-8} CPU_TYPE=${VERIFY_STALE_CPU_TYPE:-$(brev search cpu --sort price --json \ | jq -r --argjson floor "$CPU_RAM_FLOOR" \ - '[.[] | select(.stoppable == true and .ram_gb >= $floor)] | .[0].type')} + '[.[] | select(.stoppable == true and .ram_gb >= $floor)] | .[0].type // empty')} [ -n "$CPU_TYPE" ] || { echo "ERROR: no stoppable CPU SKU with >= ${CPU_RAM_FLOOR} GB RAM"; exit 1; } brev create "$INSTANCE_NAME" --type "$CPU_TYPE" fi diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/by-design.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/by-design.md index 50c4777f8fd..677567cb2df 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/by-design.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/by-design.md @@ -123,7 +123,7 @@ Use these exact link formats: - Test file: `[test/e2e/test-double-onboard.sh](https://github.com/NVIDIA/NemoClaw/blob/v0.0.35/test/e2e/test-double-onboard.sh)` - PR/issue references: bare `#NNNN` works — GitHub auto-links these in comments on the same repo, no manual URL needed. -When greping for evidence, use `git grep -n "<symbol>" "$LATEST" -- ...` so the line numbers match the tagged blob. Then construct each link from `<file path> + verified-on tag + line number`. +When grepping for evidence, use `git grep -n "<symbol>" "$LATEST" -- ...` so the line numbers match the tagged blob. Then construct each link from `<file path> + verified-on tag + line number`. The Step 8.5d self-verification pass MUST resolve at least one rendered link (e.g., `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=v0.0.35` or a `curl -fsI` to the blob URL) and confirm it returns the expected file. A broken link defeats the purpose of including the citation. If any link fails to resolve, fix it or bail to `verify-inconclusive`. diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md index 9594c3674ca..10b6ad1dbee 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md @@ -62,7 +62,7 @@ This is the version the skill will verify against. Record it — every comment m Apply these rules in order. Drop any issue that fails a rule. **Issue-type allowlist:** must have `bug` label. -**Issue-type skip:** drop if any label exactly matches `documentation`, `status: wont-fix`, `status: needs-info`, `security`, OR is `enhancement` / starts with the prefix `enhancement:` (the repo has 8 prefixed variants — `enhancement: feature`, `enhancement: MCP`, `enhancement: testing`, `enhancement: ui`, `enhancement: provider`, `enhancement: platform`, `enhancement: policy`, `enhancement: inference`, `enhancement: integration`, `enhancement: performance`, `enhancement: skill` — and exact-match misses them all; surfaced from #1752). Use the canonical repo label names — bare `wontfix` / `needs-info` are NOT the repo's labels (verified via `gh label list`); the actual labels carry a `status:` prefix and a hyphen. +**Issue-type skip:** drop if any label exactly matches `documentation`, `status: wont-fix`, `status: needs-info`, `security`, OR is `enhancement` / starts with the prefix `enhancement:` (the repo carries several prefixed variants — `enhancement: feature`, `enhancement: MCP`, `enhancement: testing`, `enhancement: ui`, `enhancement: provider`, `enhancement: platform`, `enhancement: policy`, `enhancement: inference`, `enhancement: integration`, `enhancement: performance`, `enhancement: skill` — and exact-match misses them all; surfaced from #1752; enumerate via `gh label list -L 200 --search enhancement` if you need the live set). Use the canonical repo label names — bare `wontfix` / `needs-info` are NOT the repo's labels (verified via `gh label list`); the actual labels carry a `status:` prefix and a hyphen. **Platform skip (Brev-reproducible only in v1):** drop if any of `Platform: Windows/WSL`, `Platform: MacOS`, `Platform: macOS`, `Platform: Jetson AGX Thor/Orin`. Brev has no equivalent hardware for Jetson (embedded/edge ARM with integrated GPU is not in the Brev SKU catalog), so any Brev verification of a Jetson-only bug would produce a misleading "fixed-on-x86" verdict. Keep `Platform: Ubuntu`, `Platform: DGX Spark`, `Platform: GB10`, `Platform: All`, or no platform label. `Platform: DGX Spark` and `Platform: GB10` stay in scope but Step 10 requires a "Hardware substitution" caveat in the comment naming the Brev SKU we used as a substitute (Brev x86 GPU SKUs are not faithful to GB10 / Grace Hopper silicon for performance-shape or memory-architecture-shape bugs). diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/environment-and-reproducer.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/environment-and-reproducer.md index 870e13a504a..07299a3ffc9 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/environment-and-reproducer.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/environment-and-reproducer.md @@ -52,11 +52,17 @@ The reporter's reproducer uses the <provider> provider, which requires a real AP to verify faithfully. Three options: 1. Provide an API key via file (NEVER on the command line — keys in argv are - visible in `ps -ef` to anyone with shell access on either machine). Write - the key to a 600-perm file on your laptop: - - printf '%s' '<your-key>' > ~/.nvidia-api-key + visible in `ps -ef` to anyone with shell access on either machine, and + inline `printf '...' '<key>'` leaves the key in shell history). Read the + key with a no-echo, no-history prompt, then write it to a 600-perm file + on your laptop: + + umask 077 + IFS= read -rs -p 'NVIDIA_API_KEY (input hidden): ' NVIDIA_API_KEY + printf '%s' "$NVIDIA_API_KEY" > ~/.nvidia-api-key + unset NVIDIA_API_KEY chmod 600 ~/.nvidia-api-key + echo # restore newline after the hidden read The skill copies the file to the Brev box via `brev copy` (encrypted SSH), reads it inside the box with `NVIDIA_API_KEY=$(cat ~/.nvidia-api-key)`, @@ -249,10 +255,11 @@ LOCAL_EXIT=$? echo "Local: $LOCAL_VERSION, exit $LOCAL_EXIT" ``` -Compare local result to the issue's "Actual Result" section using the same match rubric Step 8b applies on baseline: +Compare local result to the issue's "Actual Result" section using the same match rubric Step 8b applies on baseline. The two ways the predicate can fire route to different verdicts — do not collapse them: -- **Local matches the issue symptom exactly** (same exit code + same diagnostic output) AND the symptom is the post-fix expected output → skip Brev. Use the local transcript as the verified-on-latest evidence. Step 10's comment must say `Environment: local install (<version>) — Brev provisioning skipped, outcome deterministic from CLI surface alone`. -- **Local result differs from the reported "Actual Result"** → continue to Step 7 and run on Brev. The local environment may be a confound (different OS, dirty config, partial build); remote confirms. +- **Local matches the reported-bug symptom** (same exit code + same diagnostic output as the issue's "Actual Result") → route to `still-reproduces`. Use the local transcript as the verified-on-latest evidence. Step 10's comment must say `Environment: local install (<version>) — Brev provisioning skipped, bug confirmed live on latest from CLI surface alone`. +- **Local matches the expected-fixed behavior** (the symptom is gone — exit code and output are what the issue says *should* happen after the fix) → route to `fixed-on-latest`. Use the local transcript as the verified-on-latest evidence. Step 10's comment must say `Environment: local install (<version>) — Brev provisioning skipped, outcome deterministic from CLI surface alone`. +- **Local result differs from both** (neither the reported symptom nor the expected-fixed behavior) → continue to Step 7 and run on Brev. The local environment may be a confound (different OS, dirty config, partial build); remote confirms. - **Local repro errors out for environmental reasons** (`nemoclaw: command not found`, npm link broken) → continue to Step 7. Treat as inconclusive locally, not a verification failure. **If the predicate does not fire:** proceed to Step 7 normally. Most sandbox-touching bugs need Brev. diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/reproduction-rubrics.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/reproduction-rubrics.md index 0bb2acc8fa8..fbab479fe38 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/reproduction-rubrics.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/reproduction-rubrics.md @@ -82,7 +82,10 @@ LLM rewrites `./reproducer.sh` using the full issue context (description, enviro ```bash brev copy ./reproducer.sh "$INSTANCE_NAME":~/reproducer.sh -brev exec "$INSTANCE_NAME" "bash ~/reproducer.sh" 2>&1 | tee ./baseline-transcript-2.log +# Same PATH safeguard as Step 8b — non-login shells don't pick up ~/.local/bin +# automatically, and an empty PATH here misreads as `nemoclaw: command not found` +# which would route to `verify-inconclusive` for the wrong reason. +brev exec "$INSTANCE_NAME" 'export PATH="$HOME/.local/bin:$PATH" && bash ~/reproducer.sh' 2>&1 | tee ./baseline-transcript-2.log ``` - **Match:** validated (with −30 baked in). Proceed to 8d. @@ -163,10 +166,13 @@ Cross-version verification compares two moving targets: the reproducer assumes ` ```bash # Extract the primary verification command from the reproducer (e.g. "openshell forward list"). -TOOL=$(grep -oE '\b(openshell|nemoclaw)[[:space:]]+[a-z-]+' reproducer.sh | sort -u) +# Use mapfile + a quoted-array iteration so multi-word tool strings ("openshell forward") +# stay intact — bare `for t in $TOOL` word-splits them on whitespace and would pickaxe +# `openshell` and `forward` separately, weakening the drift signal. +mapfile -t TOOLS < <(grep -oE '\b(openshell|nemoclaw)[[:space:]]+[a-z-]+' reproducer.sh | sort -u) # Pickaxe each tool name across the version range. -for t in $TOOL; do +for t in "${TOOLS[@]}"; do echo "=== drift check: $t ===" git log "$REPORTED_VERSION".."$LATEST" -S"$t" --oneline -- src/ bin/ nemoclaw/src/ 2>&1 | head -5 done @@ -210,11 +216,22 @@ Performance bugs (#2598 "10s P50", #2600 "hangs ~2 min", #2733 Ollama tool-call done ``` -3. **Compute p50 and p90** for both sides. `sort -n ./latest-perf.log | awk 'NR==5'` for p50 of 10 runs. -4. **Match rubric:** - - Latest's p50 within the SLA AND baseline's p50 outside the SLA → bug fixed; same Step 9 scoring (subject to baseline-validation gate). - - Latest's p50 outside the SLA → bug still reproduces (Step 9 special case). - - Latest p50 within SLA AND baseline p50 also within SLA → reproducer doesn't actually exercise the bug; route to Step 8c synth-repro. +3. **Compute p50 and p90** for both sides. For N=10: + + ```bash + # p50 = mean of the 5th and 6th values (standard median for even N). + P50_MS=$(sort -n ./latest-perf.log | awk 'NR==5||NR==6 {sum+=$1; n++} END {printf "%.2f", sum/n}') + # p90 = 9th value (nearest-rank / NIST method for N=10). + P90_MS=$(sort -n ./latest-perf.log | awk 'NR==9') + echo "[perf] latest p50=${P50_MS}s p90=${P90_MS}s" + ``` + + Apply the same two lines to `./baseline-perf.log` for the baseline side. +4. **Match rubric (p50 fires first; p90 is the regression backstop):** + - Latest's p50 within `$SLA_P50_MS` AND baseline's p50 outside → bug fixed; same Step 9 scoring (subject to baseline-validation gate). + - Latest's p50 outside `$SLA_P50_MS` → bug still reproduces (Step 9 special case). + - Latest's p50 within `$SLA_P50_MS` AND baseline's p50 also within → reproducer doesn't actually exercise the bug; route to Step 8c synth-repro. + - **p90 backstop**: if `$SLA_P90_MS` was parsed from the issue, latest's p90 outside `$SLA_P90_MS` flips a within-SLA-p50 verdict to `still-reproduces` — tail-latency regressions matter for the issues that name them. **Hardware-substitution caveat.** Performance numbers are silicon-dependent. When the issue is `Platform: DGX Spark` or `Platform: GB10` and we're measuring on a Brev x86 GPU SKU, the comment must say so explicitly: a Brev p50 of 1.5s on a `H100` does not prove the DGX Spark p50 is fixed. Cap the score at 60 unless the bug is clearly silicon-independent (e.g. an algorithmic regression in user-space JS that would manifest the same on any silicon). diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/scoring-comments-and-logging.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/scoring-comments-and-logging.md index 13ba191d843..b536e4c33ae 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/scoring-comments-and-logging.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/scoring-comments-and-logging.md @@ -118,7 +118,7 @@ Transcripts and synth-repro scripts are already plain text and skip the pre-pass **Order matters and the patterns below are in execution order.** Longest, most-specific patterns first; generic catchalls last. Otherwise the catchall masks specific matches and you lose track of what was actually redacted (JWT vs session blob vs random base64). -Patterns live in a fenced block (not a markdown table) because patterns 8 and 9 use regex alternation `|` — markdown tables would treat the literal `|` as a column delimiter, and escaping it as `\|` makes the regex match a literal pipe instead of an alternation, which silently breaks credential redaction. +Patterns live in a fenced block (not a Markdown table) because patterns 8 and 9 use regex alternation `|` — Markdown tables would treat the literal `|` as a column delimiter, and escaping it as `\|` makes the regex match a literal pipe instead of an alternation, which silently breaks credential redaction. ```regex 1. eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,} @@ -171,7 +171,7 @@ Patterns live in a fenced block (not a markdown table) because patterns 8 and 9 | `fixed-on-latest` | **200–300 words** | Header + evidence + verdict + @-mention. Add hardware-substitution caveat or related-failure-mode section only if they shift the maintainer's read. If you're past 300, you're padding. | | `wontfix` (by-design) | **200–300 words** | Structurally-fixed + vestigial + what's-not-the-same-bug, each one to two sentences max. The PR ref carries the detail; the comment carries the verdict. | | `verify-inconclusive` | 100–200 words | One paragraph naming what the skill couldn't establish. No transcripts beyond a single quoted line. | -| **Still-reproduces (no label)** | **30–80 words** | The reporter already has the symptom; the maintainer can see the issue is open. The skill is just confirming + setting the TTL marker. **No transcripts** (the issue body has them), **no @-mention** (the reporter knows their bug is real), **no architectural prose**. One sentence stating "skill ran reproducer on `<latest>`, symptom still present" + one sentence on any partial-fix PR if relevant + marker. That's it. | +| **Still-reproduces (no label)** | **30–80 words** | The reporter already has the symptom; the maintainer can see the issue is open. The skill is just confirming + setting the TTL marker. **No transcripts** (the issue body has them), **no closing reporter @-mention** (the reporter knows their bug is real), **no architectural prose**. One sentence stating "skill ran reproducer on `<latest>`, symptom still present" + one sentence on any partial-fix PR if relevant + marker. That's it. The unanswered-question lead paragraph (rule below) is the one allowed exception when `UNANSWERED_MAINT_LOGIN` is set — it adds one maintainer @-mention as a lead, never a closing pair. | **Cut, by default:** @@ -188,7 +188,7 @@ Patterns live in a fenced block (not a markdown table) because patterns 8 and 9 **Mandatory `Verification mode` header line.** All three templates below include a `**Verification mode:**` line in the metadata block, naming what we did and didn't actually run (e.g., "runtime reproduction on Brev <SKU>; baseline + latest both installed and run" for the standard template; "static analysis at the verified-on tag — no runtime reproduction" for the by-design template; "runtime reproduction on Brev <SKU>; bug confirmed live on latest" for still-reproduces). Reader should never have to guess whether the verdict came from real install logs or from static analysis. -**Link-pass self-verification (all templates).** Same rule as Step 8.5d's link pass, applied to every template. Resolve at least one rendered markdown link from each section that has them (`What's structurally fixed` / `Vestigial references` / `Existing CI coverage` for by-design; `Relevant changes since` / transcript code-anchor citations for the standard template) via `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 if path exists at tag, 404 otherwise) or `curl -fsI <blob-url>`. A 404 on a citation in the rendered comment is worse than no citation — it advertises verification work that didn't actually happen. If any link fails to resolve, fix it or bail to `verify-inconclusive`. +**Link-pass self-verification (all templates).** Same rule as Step 8.5d's link pass, applied to every template. Resolve at least one rendered Markdown link from each section that has them (`What's structurally fixed` / `Vestigial references` / `Existing CI coverage` for by-design; `Relevant changes since` / transcript code-anchor citations for the standard template) via `gh api repos/NVIDIA/NemoClaw/contents/<path>?ref=<tag>` (returns 200 + base64 if path exists at tag, 404 otherwise) or `curl -fsI <blob-url>`. A 404 on a citation in the rendered comment is worse than no citation — it advertises verification work that didn't actually happen. If any link fails to resolve, fix it or bail to `verify-inconclusive`. **Mandatory closing block — reporter @-mention with confirmation language.** Every template below **except `Still-reproduces`** ends with an explicit @-mention of the original reporter using this exact shape: @@ -196,7 +196,7 @@ Patterns live in a fenced block (not a markdown table) because patterns 8 and 9 The skill cannot independently confirm a closed-as-fixed verdict — only the reporter knows whether their original symptom is gone in their environment. The @-mention is what converts a "skill says it's fixed" claim into actionable confirmation work for QA. Customize `<Z>` per case (the version that shipped the fix or `$LATEST`), but never omit the line. -**Mandatory unanswered-question prefix and dual @-mention.** When Step 3 sets `UNANSWERED_MAINT_LOGIN` (a maintainer's question is older than 7 days and the reporter never replied), the verdict comment changes shape in two places: +**Mandatory unanswered-question prefix and dual @-mention.** When Step 3 sets `UNANSWERED_MAINT_LOGIN` (a maintainer's question is older than 7 days and the reporter never replied), the verdict comment changes shape: 1. **Prepend a lead paragraph** as the very first line of the body, before the `## Stale-issue verification` heading. The lead paragraph is a single line: @@ -204,13 +204,15 @@ The skill cannot independently confirm a closed-as-fixed verdict — only the re [@UNANSWERED_MAINT_LOGIN's comment](UNANSWERED_MAINT_URL) from UNANSWERED_MAINT_DATE is still unanswered. Posting independent verification below to unstick the thread. ``` - …with the bracketed variables expanded from the values exported by Step 3. + …with the bracketed variables expanded from the values exported by Step 3. **Applies to all three templates** (fixed, still-reproduces, by-design). 2. **Replace the closing reporter-only @-mention with a dual @-mention** that names BOTH the maintainer (acknowledging the open question) and the reporter (per the standard confirmation pattern): > @\<UNANSWERED_MAINT_LOGIN\> — flagging that your question above is still open; the verification below may answer it. @\<reporter\> — please confirm the symptom is gone on a recent build (≥ v0.0.\<Z\>) and reopen with a fresh reproducer if you observe otherwise. -This applies to all three templates (fixed, still-reproduces, by-design). The skill becomes the *unsticking voice* on a thread that has gone quiet — never a clueless interruption when discussion is fresh (Step 3 already filtered the within-7-day case). + **Applies to `fixed-on-latest` and `by-design` only.** Still-reproduces has no closing reporter @-mention by design (see L174), so there's nothing to replace; its only nod to the unanswered maintainer is the lead paragraph from step 1. + +The skill becomes the *unsticking voice* on a thread that has gone quiet — never a clueless interruption when discussion is fresh (Step 3 already filtered the within-7-day case). **Comment template (fixed / inconclusive — bug not reproduced on latest):** @@ -262,44 +264,25 @@ This applies to all three templates (fixed, still-reproduces, by-design). The sk @<reporter> — please confirm the symptom is gone on a recent build (≥ v0.0.<Z>) and reopen with a fresh reproducer if you observe otherwise. -<!-- nemoclaw-verify-stale v1 2026-05-12 --> +<!-- nemoclaw-verify-stale v1 YYYY-MM-DD --> ```` -**Comment template (still reproduces — Step 9 special case):** +**Comment template (still reproduces — Step 9 special case).** Keep this minimal — per L174 it caps at 30–80 words, drops transcripts (issue body has them), and omits the closing reporter @-mention (the reporter knows their own bug is real). Only the unanswered-question lead paragraph (when fired) adds an @-mention; no closing dual @-mention even then. ````markdown ## Stale-issue verification — still reproducible **Reported on:** v0.0.31 -**Verified on:** v0.0.34 (commit abc1234) -**Verification mode:** runtime reproduction on Brev `<instance-class>` — baseline confirmed the symptom matches the issue; latest (v0.0.34) also produced the symptom. Bug is still live. -**Environment:** Brev <instance-class> (<instance-type>) / Ubuntu 22.04 - -The skill ran the reported reproducer on v0.0.34 and observed the same bug symptom described in this issue. The bug is still live. - -No label applied. Will re-verify automatically next weekly run; if a fix lands in the interim, the next pass catches it. - -@<reporter> — please confirm the symptom still matches your observation on v0.0.<Y> and reopen with any updated reproducer or environment details if it has shifted. - -<details><summary>Baseline transcript (validated reproducer)</summary> - -```text -<baseline transcript> -``` - -</details> - -<details><summary>Latest transcript (bug still observed)</summary> - -```text -<latest transcript> -``` +**Verified on:** v0.0.34 +**Verification mode:** runtime reproduction on Brev `<instance-class>` — bug confirmed live on latest. -</details> +Skill ran the reported reproducer on v0.0.34 and observed the same symptom. No label applied; will re-verify on the next weekly pass. -<!-- nemoclaw-verify-stale v1 2026-05-12 --> +<!-- nemoclaw-verify-stale v1 YYYY-MM-DD --> ```` +If a partial-fix PR is in flight that targets the same surface, add one sentence naming it between the verification line and the marker: `Partial fix tracked in #NNNN (not yet released).` Keep the total under 80 words. + The trailing HTML comment is the **idempotency marker** Step 3 looks for. Always include today's date in `YYYY-MM-DD` format so the candidate filter can apply the 7-day TTL. **Pre-post state-check.** A long-running verification can race with the maintainer closing the issue independently — happened on #2513 and #2519 (mid-batch closes by @jyaunches with their own verification). Re-check `state == OPEN` right before posting. If closed, apply the label tag-only (skipping the comment, since the maintainer's own close-comment is now the authoritative record) and skip the Project 199 move. From 6ba04f416da84fc45bf422d614afd28321adecf3 Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Tue, 26 May 2026 10:35:52 -0700 Subject: [PATCH 39/40] fix(verify-stale): convert p50/p90 to ms to match SLA units CodeRabbit follow-up on PR #3327: the prior fix computed P50_MS / P90_MS straight from `/usr/bin/time -f '%e'` output, which is seconds, while the match rubric compares against SLA_P50_MS / SLA_P90_MS parsed from the issue body as milliseconds. A 1.5-second p50 would have falsely beaten a 200 ms SLA. Multiply by 1000 in the awk, format as integer ms, and update the echo suffix accordingly. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../reference/reproduction-rubrics.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/reproduction-rubrics.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/reproduction-rubrics.md index fbab479fe38..cf64ff6f0e0 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/reproduction-rubrics.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/reproduction-rubrics.md @@ -216,17 +216,21 @@ Performance bugs (#2598 "10s P50", #2600 "hangs ~2 min", #2733 Ollama tool-call done ``` -3. **Compute p50 and p90** for both sides. For N=10: +3. **Compute p50 and p90** for both sides, in milliseconds (to match the `_MS` + units of `SLA_P50_MS` / `SLA_P90_MS`). `/usr/bin/time -f '%e'` emits + seconds, so multiply by 1000 in the awk: ```bash - # p50 = mean of the 5th and 6th values (standard median for even N). - P50_MS=$(sort -n ./latest-perf.log | awk 'NR==5||NR==6 {sum+=$1; n++} END {printf "%.2f", sum/n}') - # p90 = 9th value (nearest-rank / NIST method for N=10). - P90_MS=$(sort -n ./latest-perf.log | awk 'NR==9') - echo "[perf] latest p50=${P50_MS}s p90=${P90_MS}s" + # p50 = mean of the 5th and 6th values (standard median for even N), in ms. + P50_MS=$(sort -n ./latest-perf.log \ + | awk 'NR==5||NR==6 {sum+=$1; n++} END {printf "%d", (sum/n)*1000}') + # p90 = 9th value (nearest-rank / NIST method for N=10), in ms. + P90_MS=$(sort -n ./latest-perf.log | awk 'NR==9 {printf "%d", $1*1000}') + echo "[perf] latest p50=${P50_MS}ms p90=${P90_MS}ms" ``` - Apply the same two lines to `./baseline-perf.log` for the baseline side. + Apply the same two lines to `./baseline-perf.log` for the baseline side + (export as `BASELINE_P50_MS` / `BASELINE_P90_MS`). 4. **Match rubric (p50 fires first; p90 is the regression backstop):** - Latest's p50 within `$SLA_P50_MS` AND baseline's p50 outside → bug fixed; same Step 9 scoring (subject to baseline-validation gate). - Latest's p50 outside `$SLA_P50_MS` → bug still reproduces (Step 9 special case). From 59aa03f7b0c9c9277837f983e06c3c11a603911f Mon Sep 17 00:00:00 2001 From: Prekshi Vyas <prekshiv@nvidia.com> Date: Tue, 26 May 2026 10:40:36 -0700 Subject: [PATCH 40/40] fix(verify-stale): v-prefix normalization + readable question-detection regex MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two CodeRabbit follow-ups on PR #3327's candidate-selection.md: - Step 4 tag validation now normalizes each candidate to the full tag form (prepend `v` when absent) before `grep -Fxq` against the tag list. The body/comment capture group is digits-only by design, but tags carry the `v` — without the prepend, every body-sourced candidate was being dropped. Label-sourced candidates already carry the `v`, so the prepend is idempotent. - Split the long alternation regex behind the unanswered-maintainer question detector into four named test() clauses (literal "?", polite imperative, modal interrogative, "do you ..."). Future heuristics drop in as a single test() append rather than a regex patch. Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> --- .../reference/candidate-selection.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md index 10b6ad1dbee..f19252141db 100644 --- a/.agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md +++ b/.agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md @@ -111,11 +111,17 @@ Run this check for every candidate that survived the label-based filters above; REPORTER=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json author --jq .author.login) # Most recent unanswered maintainer comment that looks like a question — filters out triage acknowledgments (#1642 surfaced this). +# Question-detection patterns are chained as separate test() calls so each +# heuristic is independently readable and a future addition (e.g. "what about", +# "why does") is a one-line append rather than a regex-alternation patch. UNANSWERED_MAINT=$(gh issue view "$ISSUE_NUMBER" --repo NVIDIA/NemoClaw --json comments \ --jq --arg reporter "$REPORTER" --arg cutoff "$SEVEN_DAYS_AGO" ' (.comments | map(select((.authorAssociation == "MEMBER" or .authorAssociation == "OWNER" or .authorAssociation == "COLLABORATOR") - and (.body | test("\\?|(?i)\\bplease (confirm|share|provide|clarify|tell|verify|check|let me know|let us know)|(?i)\\b(could|can|would) you\\b|(?i)\\bdo you (have|know|see|use)\\b")))) + and (.body | test("\\?") # literal "?" + or test("(?i)\\bplease (confirm|share|provide|clarify|tell|verify|check|let me know|let us know)") # polite imperative + or test("(?i)\\b(could|can|would) you\\b") # modal interrogative + or test("(?i)\\bdo you (have|know|see|use)\\b")))) # "do you ..." | sort_by(.createdAt) | last) as $maint | if $maint == null then null else @@ -182,10 +188,15 @@ Collect every match from sources 2 and 3 (a single body may mention multiple ver - Future roadmap labels that slipped past source 1. - Versions parsed from prose that happen to look semver-ish but aren't releases. +**Normalize to tag form before validating.** The body/comment regex captures only the digit portion (`(\d+\.\d+\.\d+)`) — the leading `v?` sits outside the capture group on purpose. Tags carry the `v`, labels carry the `v`, and `REPORTED_VERSION` (set on L196 below) must carry the `v`. Without an explicit prepend, `grep -Fxq "0.0.32"` against a tag list whose entries are `v0.0.32` would drop every body-sourced candidate. + ```bash gh api repos/NVIDIA/NemoClaw/tags --paginate --jq '.[].name' > /tmp/nemoclaw-tags.txt -# For each candidate version V: +# For each candidate version V — normalize to full tag form, then validate. +# Label-sourced candidates already have the `v` (idempotent); body/comment-sourced +# candidates do not. +[[ "$V" =~ ^v ]] || V="v$V" grep -Fxq "$V" /tmp/nemoclaw-tags.txt || drop_version "$V" ```