diff --git a/.agents/skills/nemoclaw-maintainer-ci-flake-triage/MULTI-MODEL-TESTING.md b/.agents/skills/nemoclaw-maintainer-ci-flake-triage/MULTI-MODEL-TESTING.md new file mode 100644 index 00000000000..f88bcf69e48 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-ci-flake-triage/MULTI-MODEL-TESTING.md @@ -0,0 +1,27 @@ +# Multi-model test plan — ci-flake-triage + +## Models in scope + +| Model | Check | +|---|---| +| Claude Haiku 4.5 | Does Haiku correctly fetch the failing log tail and parse the test name? | +| Claude Sonnet 4.6 | Does Sonnet apply diff-overlap + main-history checks in the documented order? | +| Claude Opus 4.7 (1M) | Does Opus auto-elevate chronic flakes correctly when the history threshold fires? | + +## Pass criteria + +- Failing test correctly identified from log; file:line surfaced +- Diff-overlap check uses git diff --name-only origin/main..PR_HEAD (not the PR's branch alone) +- Main-history check uses gh run list --workflow= --branch=main --limit=5 +- Auto-elevation to INFRASTRUCTURE fires at >=7 hits in 14 days (read from /tmp/flake-history.jsonl) +- Infrastructure signatures matched verbatim (docker daemon, 502/503, ENETUNREACH, runc create failed) + +## Known risks + +- Haiku may skip the multi-rerun check if there's only one failed run; document the fallback ("with one rerun, lean on diff-overlap"). +- Sonnet may forget to append to /tmp/flake-history.jsonl. Make the append step a hard rule. +- Opus might over-investigate (read 500 lines of log); enforce the --fetch-logs 50KB max. + +## How to run + +Synthesize 3 failing-check scenarios (PR_CAUSED, PRE_EXISTING_FLAKE, INFRASTRUCTURE) and verify each verdict. diff --git a/.agents/skills/nemoclaw-maintainer-ci-flake-triage/SKILL.md b/.agents/skills/nemoclaw-maintainer-ci-flake-triage/SKILL.md new file mode 100644 index 00000000000..e8bfa5ef598 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-ci-flake-triage/SKILL.md @@ -0,0 +1,172 @@ +--- +name: nemoclaw-maintainer-ci-flake-triage +description: Classifies a failing CI check as PR_CAUSED / PRE_EXISTING_FLAKE / INFRASTRUCTURE / UNCLEAR. Combines diff-overlap detection, recent main-history comparison, multi-rerun reproducibility analysis, and infrastructure-signature greps to produce a verdict with evidence in ~2 min instead of ~12 min of manual log digging. Tracks flake history at /tmp/flake-history.jsonl and auto-elevates chronic flakes (7+ hits in 14 days) to INFRASTRUCTURE. Use when a PR's CI is red and the question is "is this me?" before going deeper on a fix. Local-only. +--- + +# CI Flake Triage + +When a PR shows red CI, the maintainer's first 5 minutes are deciding: "is this MY fault, or is the test broken?" This skill answers that, fast, with evidence. + +## Why this matters + +In the 2026-05-13/14 session, multiple PR runs surfaced failures that turned out to be unrelated to the diff: + +- PR #3409: `macos-e2e` failing on `test/cli.test.ts:1310` (logs/debug test) — pre-existing flake on main HEAD `edb7478a2` +- PR #3501: `test-e2e-port-overrides` failing on Test #4 first run, Test #6 second run — same code, different sub-tests = nondeterministic flake +- PR #3498 etc.: pre-existing 5s testTimeout flakes in `onboard-probes`, `dns-proxy`, `cli`, `repro-2666`, `onboard` + +Without a fast triage step, a maintainer spends ~15 min per red check chasing a ghost. This skill collapses that to ~2 min with a clear verdict. + +## Invocation + +```text +/nemoclaw-maintainer-ci-flake-triage [check-name] +``` + +If `check-name` is omitted, the skill picks the first FAILURE in the PR's status check rollup. + +Flags: + +| Flag | Default | Meaning | +|------|---------|---------| +| `--main-history N` | `5` | How many recent main runs to compare against | +| `--time-budget-min` | `2` | Pause + ask if triage hasn't reached a verdict | +| `--fetch-logs` | `on` | Pull the failing-step log (50KB max) for the verdict | + +## Verdict taxonomy + +| Verdict | Definition | Recommended action | +|---|---|---| +| `PR_CAUSED` | The diff touches code reachable by the failing test, AND the same test passes on recent main, AND the failure message matches a regression pattern. | Fix the PR — drop into Stage 4 of `issue-autopilot` if a clear fix is obvious, otherwise ask user. | +| `PRE_EXISTING_FLAKE` | The failing check has failed on at least one of the last N main runs OR the same test fails on this PR rerun-fresh with no code change. | Note the flake in the PR comments, do NOT block on it. | +| `INFRASTRUCTURE` | The error is platform-runtime (Docker daemon, runner network, GH API 502, dist not present), not a test assertion failure. | Rerun the check; if it recurs, surface to platform owners. | +| `UNCLEAR` | Insufficient signal — log truncated, test name doesn't map to diff cleanly, main history is short. | Halt, surface evidence, ask user. | + +## Workflow + +1. **Fetch the PR's check rollup.** `gh pr view --json statusCheckRollup`. +2. **Pick the failing check.** Explicit `check-name` argument, or first `conclusion=FAILURE`. +3. **Get the failing-step log tail.** `gh run view --log-failed | tail -100`. +4. **Identify the failing test/assertion.** Parse the log for `FAIL:`, `Error:`, `AssertionError`, `expect(...).toBe(...)`, `Test timed out`, etc. Extract the test name + file:line. +5. **Diff-overlap check.** `git diff --name-only origin/main..PR_HEAD` — does the test file (or files the test imports / files in the same package) appear in the diff? + - YES + clear reachability → leans `PR_CAUSED` + - NO overlap → leans `PRE_EXISTING_FLAKE` or `INFRASTRUCTURE` +6. **Main-history check.** For the same workflow/job/check, look at the last `--main-history N` main runs: + - Same check failed on main recently → strong `PRE_EXISTING_FLAKE` signal + - Same check passing consistently on main → strong `PR_CAUSED` signal (assuming step 5 also points there) + + ```bash + gh run list --workflow= --branch=main --limit=N --json conclusion,headSha,createdAt + ``` + +7. **Multi-run reproducibility.** If the PR has multiple recent reruns, check whether different tests fail across reruns: + - Different tests failing each rerun → nondeterministic flake (`PRE_EXISTING_FLAKE` even if not on main yet — the test fixture is unstable). +8. **Infrastructure signature check.** Grep the log for: + - `cannot connect to docker daemon` / `Docker daemon not running` + - `502 Bad Gateway` / `503 Service Unavailable` + - `ENETUNREACH` / `ETIMEDOUT` on github.com + - `Module ... has no exported member` (stale dist; rebuild) + - `runc create failed` +9. **Emit verdict.** A short paragraph + evidence table + recommended action. +10. **Append to flake history.** See "Flake history database" section below. + +## Flake history database (hardening — auto-elevate chronic flakes) + +Every PRE_EXISTING_FLAKE and INFRASTRUCTURE verdict appends one line to `/tmp/flake-history.jsonl`: + +```json +{"ts":"2026-05-14T...","run_id":"...","pr":3501,"check_name":"test-e2e-port-overrides","verdict":"PRE_EXISTING_FLAKE","failing_test":"Test #4 ...","main_history_hit":false} +``` + +Before emitting a final verdict, query the history for recent hits on the same `check_name`: + +```bash +RECENT_HITS=$(jq -c "select(.check_name == \"$CHECK_NAME\" and (.ts | fromdateiso8601) > (now - 1209600))" /tmp/flake-history.jsonl | wc -l | tr -d ' ') +if [ "$RECENT_HITS" -ge 7 ]; then + echo " ⚠ $CHECK_NAME has been flagged $RECENT_HITS times in the last 14 days" + echo " AUTO-ELEVATE: PRE_EXISTING_FLAKE → INFRASTRUCTURE" + echo " RECOMMENDED ACTION: file a fix-up issue tagged 'ci-infra' for this check" +fi +``` + +**Elevation rule:** + +- ≥7 hits in 14 days → auto-elevate `PRE_EXISTING_FLAKE` → `INFRASTRUCTURE` and prepend an action to file a tracking issue. +- ≥3 hits in 7 days → keep the original verdict but add a "consider filing a flake-fix issue" recommendation. + +The auto-filed issue draft template: + +```markdown +**Title:** ci: stabilize chronic flake in `` + +**Body:** +This check has been flagged as flaky 7+ times in the last 14 days (see /tmp/flake-history.jsonl, runs ). + +Pattern: +Failing test(s): +Latest run: + +Action: investigate the underlying nondeterminism in the test fixture or runner setup. PRs blocked on this check should not be held up. +``` + +The history file is local-only by default; never committed. It survives between runs because `/tmp` typically persists across shell sessions on macOS/Linux; if the maintainer wants persistent history across reboots, move it to `~/.nemoclaw/flake-history.jsonl`. + +## JSON sidecar output + +Writes `/tmp/nemoclaw-skill-output-ci-flake-triage-.json`. Consumed by `issue-autopilot` Stage 7 (CI watch) to decide whether to halt or proceed. + +**Envelope:** shared maintainer-skill schema (see `find-already-fixed/SKILL.md`). + +**Per-result shape:** + +```json +{ + "pr": 3501, + "check_name": "test-e2e-port-overrides", + "check_url": "https://...", + "verdict": "PR_CAUSED" | "PRE_EXISTING_FLAKE" | "INFRASTRUCTURE" | "UNCLEAR", + "confidence": "high" | "medium" | "low", + "evidence": { + "failing_test": "Test #4 \"Privileged port 80 rejected by entrypoint\"", + "diff_overlap": false, + "main_history": [{"sha": "...", "conclusion": "success", "createdAt": "..."}], + "multi_rerun_pattern": "different-tests-different-reruns" + }, + "recommended_action": "Add PR comment noting flake; do NOT block on this check", + "draft_pr_comment": null +} +``` + +If verdict is `PRE_EXISTING_FLAKE` or `INFRASTRUCTURE`, callers should NOT halt on the failure. If `PR_CAUSED`, callers must address it. `UNCLEAR` halts and asks the user. + +## Output discipline + +```text +=== CI flake triage: PR #3501 / test-e2e-port-overrides === + +VERDICT: PRE_EXISTING_FLAKE (high confidence) + +Evidence: +- Failing test: Test #4 "Privileged port 80 rejected by entrypoint" (e2e-port-overrides.sh:104-110) +- PR diff: only docs/* files touched — zero overlap with port-validation code +- Main history: gpu-e2e_25842713195 (today, success), 25843401869 (today, success), prior 5 runs all success +- Multi-rerun reproducibility: Test #4 failed on commit 1771ce6c2, Test #6 failed on commit d1b9a4115 — different tests, same script, same source = nondeterministic stderr capture in fixture + +Recommended action: +- Add a one-line PR comment noting the flake and link to a separate flake-triage issue +- File `test/e2e-port-overrides.sh:107` stderr-capture race as a follow-up issue +- Do NOT block PR #3501 on this check +``` + +## Halt conditions + +- Verdict reached → emit and stop. +- `UNCLEAR` after 2-min budget → halt; this is usually a log-truncation case where the failure-message parser couldn't pin the failing test. + +## Hard nos + +- Verdict-only. No flake fixes, no CI reruns, no PR comments posted. The `recommended_action` field tells the caller what to do; the caller decides. + +## Reference run + +PR #3501 wsl-e2e and macos-e2e both passed; only `test-e2e-port-overrides` failed across 2 reruns on docs-only commits. Verdict `PRE_EXISTING_FLAKE` was reached in <60s via main-history check + diff-overlap check + multi-rerun reproducibility. Canonical happy path. diff --git a/.agents/skills/nemoclaw-maintainer-ci-flake-triage/evals/01.json b/.agents/skills/nemoclaw-maintainer-ci-flake-triage/evals/01.json new file mode 100644 index 00000000000..ed394fc4c74 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-ci-flake-triage/evals/01.json @@ -0,0 +1,13 @@ +{ + "skills": [ + "nemoclaw-maintainer-ci-flake-triage" + ], + "query": "PR #3501's test-e2e-port-overrides is red \u2014 is this my fault?", + "files": [], + "expected_behavior": [ + "Fetches the failing-step log tail; parses the failing test name + file:line.", + "Runs diff-overlap check: git diff --name-only origin/main..PR_HEAD vs the test's source files.", + "Runs main-history check: gh run list --workflow= --branch=main --limit=5 to see if the same check has been failing recently.", + "If diff-overlap=false AND main-history shows failures: verdict=PRE_EXISTING_FLAKE with confidence=high; recommends commenting the flake on the PR but NOT blocking on it." + ] +} diff --git a/.agents/skills/nemoclaw-maintainer-ci-flake-triage/evals/02.json b/.agents/skills/nemoclaw-maintainer-ci-flake-triage/evals/02.json new file mode 100644 index 00000000000..321236f02ee --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-ci-flake-triage/evals/02.json @@ -0,0 +1,15 @@ +{ + "skills": [ + "nemoclaw-maintainer-ci-flake-triage" + ], + "query": "test-cli has flaked 8 times in the last 2 weeks. Should it be elevated?", + "files": [ + "/tmp/flake-history.jsonl" + ], + "expected_behavior": [ + "Queries /tmp/flake-history.jsonl for prior PRE_EXISTING_FLAKE entries with check_name == 'test-cli' in last 14 days.", + "When count >= 7 -> auto-elevates current verdict from PRE_EXISTING_FLAKE to INFRASTRUCTURE.", + "Drafts a recommended action: file a fix-up issue tagged 'ci-infra' for this check, with the latest run URL and the multi_rerun_pattern observation.", + "Appends the current verdict to /tmp/flake-history.jsonl for future calibration." + ] +} diff --git a/.agents/skills/nemoclaw-maintainer-ci-flake-triage/evals/03.json b/.agents/skills/nemoclaw-maintainer-ci-flake-triage/evals/03.json new file mode 100644 index 00000000000..73b47daa0c7 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-ci-flake-triage/evals/03.json @@ -0,0 +1,13 @@ +{ + "skills": [ + "nemoclaw-maintainer-ci-flake-triage" + ], + "query": "The failing log shows 'cannot connect to docker daemon'. What's the verdict?", + "files": [], + "expected_behavior": [ + "Runs the infrastructure-signature grep: matches 'cannot connect to docker daemon' as a known INFRASTRUCTURE pattern (along with 502/503 Bad Gateway, ENETUNREACH/ETIMEDOUT, runc create failed, stale dist errors).", + "Verdict: INFRASTRUCTURE with confidence=high regardless of diff-overlap or main-history.", + "Recommended action: rerun the check; if it recurs, surface to platform owners.", + "Does NOT recommend a code fix on the PR side." + ] +}