fix(e2e): stabilize release candidate validation

[cjagwani]

Summary

Fix the non-Jetson failures from the v0.0.82 release-candidate run. Managed recovery now distinguishes transient supervisor races from definitive failure, the GPU and full-E2E assertions use bounded evidence-based limits, and the Hermes rebuild fixture reclaims only its exact owned images instead of accumulating them until the hosted runner is lost.

Changes

• Retry only exact retryable managed-supervisor results after the recovery settle window, keep definitive failures authoritative, and validate gateway stability through the supervisor PID/starttime record.
• Give only the post-build Docker stop operation a 90-second deadline and keep the GPU PONG assertion out of the model reasoning-token budget.
• Record four functional post-image-change full-E2E observations and derive bounded adjustments for only the root-start and sandbox-phase caps.
• Preserve the exact OpenShell-derived Hermes image tag in curated registry state, reclaim the initial and old fixture images by validated tag, release the temporary old-base tag, and assert rebuild removed the old derived tag.
• Add focused regression coverage and E2E mock-parity ownership for the escaped failure classes.

Escaped-defect evidence

The exact-candidate run exposed test assumptions that existing fast coverage did not exercise: retryable post-settle supervisor responses were treated as definitive, the live PID oracle parsed process listings instead of the supervisor identity record, a routed GPU response could spend its small token budget on reasoning, and the Hermes fixture overwrote registry image ownership before production cleanup could reclaim it. The Hermes jobs then lost separate hosted runners before logs or artifacts could upload. The new focused tests cover the stable local boundaries, while the live fixture now records and proves exact image reclamation; a fresh full default E2E run remains the authoritative integrated validation after merge.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent correctness and security review found no actionable findings; managed recovery remains fail-closed, destructive image cleanup requires exact fixture-owned tags, and no broad prune or self-hosted runner migration was introduced.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

• PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npm run test:changed passed 1,041 tests in 102 files; focused Hermes ownership/progress and mock-parity tests passed; CLI typecheck, exact project membership, source-shape guard, Biome, and diff checks passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — not claimed: an exploratory local npm test was stopped after unrelated local concurrency/environment failures; required PR CI and the fresh exact-SHA full live run are the authoritative broad evidence.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Charan Jagwani cjagwani@nvidia.com

Summary by CodeRabbit

• Bug Fixes

  • Improved managed gateway recovery: temporary probe states are treated as indeterminate, and settle-window verification now uses a controlled two-check confirmation before declaring success or failure.
  • Reduced false gateway PID matches by validating supervisor PID records against /proc start-identity/state.
  • Increased the GPU compatibility container stop timeout (stop provisioning container phase) to improve reliability.

• Tests / Validation

  • Updated e2e and controller recovery coverage for settle-deadline behavior, refreshed Hermes rebuild tag parsing/cleanup assertions, and adjusted GPU inference request payload.
  • Refined onboarding performance budget/calibration expectations and schema contract strings, including bounded cold-path validation cap adjustments.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The PR updates managed gateway recovery retries, adds a dedicated GPU patch stop timeout, strengthens E2E gateway PID and Hermes image-state handling, adjusts GPU inference test parameters, and records validation-derived cold-path performance budgets.

Changes

Cold-path performance calibration

Layer / File(s)	Summary
Calibration data and budgets ci/full-e2e-cold-path-calibration.json, ci/onboard-performance-budget.json, ci/source-shape-test-budget.json	Calibration metadata records four validation runs and derives updated root-start and sandbox caps; corresponding budget text, values, and source-shape exceptions are updated.
Calibration schema validation test/onboard-performance-config-schema.test.ts	Calibration types, threshold helpers, effective-budget calculations, and assertions validate the adjustment structure and checked-in budgets.

Managed gateway recovery

Layer / File(s)	Summary
Recovery probe retry behavior src/lib/actions/sandbox/process-recovery.ts	Retryable managed failures remain indeterminate, and settle-phase managed probes run before and at the deadline.
Recovery race coverage test/process-recovery-managed-controller.test.ts, src/lib/actions/sandbox/process-recovery.test.ts, src/lib/actions/sandbox/gateway-restart.test.ts	Tests cover transient supervisor failures, persistent busy responses, sequential managed probe results, and updated settle-window call timing.

Compatibility GPU patch timeout

Layer / File(s)	Summary
Dedicated container stop timeout src/lib/onboard/docker-gpu-patch-constants.ts, src/lib/onboard/docker-gpu-patch-recreate.ts, src/lib/onboard/docker-gpu-patch-recreate.test.ts	Container recreation uses a dedicated 90-second stop timeout, with a matching test assertion.

E2E fixture reliability

Layer / File(s)	Summary
Supervisor-owned gateway PID validation test/e2e/fixtures/clients/gateway.ts, test/e2e/support/e2e-recovery-helpers.test.ts	Gateway PID resolution validates the supervisor PID record and procfs start identity; observation duration validation and helper tests are strengthened.
Hermes image-state tracking test/e2e/live/rebuild-hermes-image-state.ts, test/e2e/live/rebuild-hermes.test.ts	Hermes image tags are parsed and preserved through registry seeding, then removed through tag-specific cleanup checks.
E2E validation coverage test/e2e/support/rebuild-hermes-image-state.test.ts, test/e2e/mock-parity.json, test/e2e/live/gpu-e2e.test.ts	Image-state parsing receives focused tests and mock parity wiring; GPU inference requests set reasoning_effort to none.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RecoveryController
  participant ManagedProbe
  participant SettleWindow
  RecoveryController->>ManagedProbe: Probe before settle deadline
  ManagedProbe-->>RecoveryController: Return managed health result
  RecoveryController->>SettleWindow: Sleep until deadline
  RecoveryController->>ManagedProbe: Probe at settle deadline
  ManagedProbe-->>RecoveryController: Return final managed health result

Loading

Possibly related PRs

• NVIDIA/NemoClaw#5809: Both changes modify the Hermes rebuild E2E harness and curated registry/session image-state handling.

Suggested labels: area: e2e, bug-fix, area: sandbox

Suggested reviewers: cv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and clearly matches the PR’s main goal of stabilizing release-candidate E2E validation.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-release-e2e-evidence-followups

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage remains at 96%, unchanged from the main branch.

TypeScript / code-coverage/cli

The overall coverage in the codex/fix-release-e2... branch remains at 80%, unchanged from the main branch.

Show a code coverage summary of the most impacted files.

File	main 0ec7e97	codex/fix-release-e2... ea03ee1	+/-
src/lib/securit...ntial-filter.ts	99%	98%	-1%
src/lib/onboard...ch-constants.ts	100%	100%	0%
src/lib/actions...ess-recovery.ts	82%	84%	+2%
src/lib/agent/aliases.ts	89%	94%	+5%

---

_{Updated July 15, 2026 19:55 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.}

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, gpu-e2e, onboard-repair, onboard-resume, rebuild-hermes, rebuild-hermes-stale-base

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/actions/sandbox/process-recovery.ts`:
- Around line 791-800: Update the managed probe retry loop around managedProbe
so its retry count or deadline is derived from settleSeconds, not the broader
attempts budget based on timeoutSeconds. Ensure retries, including interval
sleeps, cannot extend beyond the configured settle window while preserving the
existing authenticated-result and definitive-failure behavior.

In `@test/e2e/support/e2e-recovery-helpers.test.ts`:
- Around line 196-200: Replace the literal shell-script assertions in the
resolveGatewayPid test with behavior-based cases through the public
resolveGatewayPid boundary. Provide scripted process responses that demonstrate
matching starttime acceptance and stale or mismatched PID rejection, and assert
the returned outcomes; remove the toContain/not.toContain checks for
implementation details.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f51ca997-1cbb-43e0-be8e-0279a89fd624

📥 Commits

Reviewing files that changed from the base of the PR and between ac97fdc and ab6d726.

📒 Files selected for processing (16)

• ci/full-e2e-cold-path-calibration.json
• ci/onboard-performance-budget.json
• ci/source-shape-test-budget.json
• src/lib/actions/sandbox/process-recovery.ts
• src/lib/onboard/docker-gpu-patch-constants.ts
• src/lib/onboard/docker-gpu-patch-recreate.test.ts
• src/lib/onboard/docker-gpu-patch-recreate.ts
• test/e2e/fixtures/clients/gateway.ts
• test/e2e/live/gpu-e2e.test.ts
• test/e2e/live/rebuild-hermes-image-state.ts
• test/e2e/live/rebuild-hermes.test.ts
• test/e2e/mock-parity.json
• test/e2e/support/e2e-recovery-helpers.test.ts
• test/e2e/support/rebuild-hermes-image-state.test.ts
• test/onboard-performance-config-schema.test.ts
• test/process-recovery-managed-controller.test.ts

[cjagwani]

Addressed both informational advisor warnings in 91c42396d755b9d90f2211cc6e7eeb79084e3edf:

• PRA-1: documented the controller-owned source states, bounded host-side rationale, and concrete removal condition; added a transient-marker-then-terminal-result case proving terminal results stop retries.
• PRA-2: added an explicit five-successful-sample, single-exact-head retirement trigger for the interim adjustment and verified that removing the adjustment restores the exact-head baseline budgets.

Focused verification: 28 targeted integration tests, 491 changed-scope tests, source-shape guard, Biome, diff checks, and npm run check:diff all pass.

[cjagwani]

Addressed both current CodeRabbit findings in 1a653e8:

• Managed gateway settle confirmation is now bounded by settleSeconds, including the final deadline probe, while preserving authenticated-success and definitive-failure behavior.
• Gateway PID validation is now exercised behaviorally through resolveGatewayPid, including matching supervisor identity acceptance and stale/reused PID rejection.

Validation on this head: focused recovery suites, 830 affected tests via npm run test:changed, source-shape checks, git diff --check, and npm run check:diff all pass. The commit is GitHub Verified.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/fixtures/clients/gateway.ts`:
- Around line 185-186: Update the shell validation in expectPidStable to inspect
the process state from $1 after set -- $rest and exit before printing the
identity when it is a terminal state such as Z or X. Preserve the existing
start-time validation and only print the PID identity for non-terminal
processes.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 014367e9-8c20-4d21-a6dc-dc94c9c15a3b

📥 Commits

Reviewing files that changed from the base of the PR and between 91c4239 and 1a653e8.

📒 Files selected for processing (6)

• src/lib/actions/sandbox/gateway-restart.test.ts
• src/lib/actions/sandbox/process-recovery.test.ts
• src/lib/actions/sandbox/process-recovery.ts
• test/e2e/fixtures/clients/gateway.ts
• test/e2e/support/e2e-recovery-helpers.test.ts
• test/process-recovery-managed-controller.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• test/process-recovery-managed-controller.test.ts

[wscurran]

✨ Thanks for the fix, @cjagwani. Stabilizing the release candidate validation should improve daily release reliability. Ready for maintainer review.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-6840.docs.buildwithfern.com/nemoclaw

[cjagwani]

Superseded — do not authorize this request. main advanced before authorization, so head b677a986e5cdf81e012c364d2e5ce0728d91f0ba, base 3461d7186ae9272571b54e5de5a7f79ca67152ef, and plan 85a2eee509bc549fb7f7b0929c128822ff1680a029837b621d8d6c40e0c830bd are stale.

PR #6840 is now conflict-free at head ba11da6a975ee7729daffc1c490d31d22d76d709 on base ea143b817d1c28223e6a997de97276776d4ea32d. Fresh CI and review are running; a new exact E2E authorization request will follow after the planner publishes it.

[cv]

Maintainer pass on exact head ba11da6a975ee7729daffc1c490d31d22d76d709: normal CI is green, but this is not ready for live-E2E authorization yet.

The current primary advisor warning is still valid. validationAdjustment raises the enforced root-start and sandbox caps from four historical runs, while test/onboard-performance-config-schema.test.ts validates only SHA shape, run fields, and cap arithmetic. It does not prove that every run head descends from imageChangeSha or that the declared image-building inputs stayed unchanged through imageInputsVerifiedThroughSha. I manually confirmed that the four recorded run/job conclusions and current ancestry claims are consistent, but that one-time review does not protect the checked-in relaxation from later stale or unrelated evidence.

Please either add a repository-backed provenance assertion for the adjustment evidence (including a negative case), or replace the adjustment with a fresh exact-head five-run calibration and remove the interim claims. Also sync the branch with current main (45b1cb5a01f3408743c8479772af4ad9d4a91fc8) before requesting authorization so normal CI/review and any E2E receipt are tied to the current base.

@coderabbitai resume

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

#!/bin/bash
set -euo pipefail
fd -a 'onboard-performance-config-schema.test.ts' || true
fd -a 'full-e2e-cold-path-calibration.json' || true
echo "---calibration json---"
cat ci/full-e2e-cold-path-calibration.json 2>/dev/null | head -200
echo "---schema test---"
sed -n '1,250p' test/onboard-performance-config-schema.test.ts 2>/dev/null
</parameter>
</invoke>

[cjagwani]

@cv, please authorize the credentialed E2E plan for this exact reviewed revision.

• PR: 6840
• head: 20644fb5b9b0951465124c85515a8b2d6078d7de
• base: 88f2dd8b5d53cb6b499b6dd43dd66d7c3a704924 (still current main)
• deterministic plan: 63fe385c62842945f5055abf9f4e56e3eb16af124caf3240e4cb67bd85679199
• selected jobs: cloud-onboard, credential-sanitization, security-posture, gpu-e2e, onboard-repair, onboard-resume, rebuild-hermes, rebuild-hermes-stale-base
• ordinary CI: 46 successful checks, 0 ordinary failures
• automated review: exact-head canonical advisor 0 blockers / 0 warnings / 0 suggestions; Terra 0/0/0 and Nemotron 0/0/0, high confidence
• CodeRabbit: all historical threads resolved and outdated; no active unresolved thread
• compliance: sync and follow-up commits are GitHub Verified and DCO-signed
• local review/validation: independent security/correctness PASS; focused post-sync suites 41/41 and check:diff passed

The coordination failure is the expected authorization gate: no selected E2E job ran and no repository secret was exposed. Please run run-control-plane in the E2E / PR Gate Controller with the exact PR/head/base above and a review reason. Plans 96819877… and cef7147d… are stale and must not be used.

[cjagwani]

@jyaunches, could you authorize the exact credentialed E2E plan while @cv is unavailable?

• head: 20644fb5b9b0951465124c85515a8b2d6078d7de
• base/current main: 88f2dd8b5d53cb6b499b6dd43dd66d7c3a704924
• plan: 63fe385c62842945f5055abf9f4e56e3eb16af124caf3240e4cb67bd85679199
• selected jobs: cloud-onboard, credential-sanitization, security-posture, gpu-e2e, onboard-repair, onboard-resume, rebuild-hermes, rebuild-hermes-stale-base

All ordinary CI and exact-head Terra/Nemotron reviews are clean; CodeRabbit has no active unresolved thread; commits are Verified and DCO-signed. No credentialed job has run and no secret was exposed. Please use run-control-plane with PR 6840 and the exact head/base above. The older 96819877… and cef7147d… plans remain stale.

[cjagwani]

SUPERSEDED — do not authorize or reuse this plan. The referenced ecbedd9…/051bb4e… lifecycle was superseded and its child run was canceled. A fresh exact 795341c…/627154d… request will follow after current CI produces the new deterministic plan.

@cv @jyaunches — please review and authorize the exact credentialed E2E plan for #6840 through the E2E / PR Gate Controller run-control-plane workflow.

• Head: ecbedd966fb41bf6b3311dced0eaedc576cd9d16
• Base/current main: 051bb4e3a81c6f586cc2d5f10061eecd95520be9
• Deterministic plan: ca20ea736005f97f6aceeefe640218ac6f55936294c9f05198d2e5c9c19e4e37
• Coordination check: https://github.com/NVIDIA/NemoClaw/runs/87410177723
• Suggested review reason: Exact-head release stabilization validation, including Hermes rebuild image cleanup and stale-base recovery.

Selected jobs:

• cloud-onboard
• credential-sanitization
• security-posture
• gpu-e2e
• onboard-repair
• onboard-resume
• rebuild-hermes
• rebuild-hermes-stale-base

The PR is open, mergeable, approved, and bound to current main. The head commit is GitHub Verified and DCO-signed. Exact-head Terra and Nemotron are both high-confidence 0/0/0; CodeRabbit is successful with zero active unresolved threads; required PR CI is green with no failures. The plan confirms that no selected job has run and no repository secret has been exposed.

Please enter PR 6840, the exact head/base SHAs above, and the review reason in run-control-plane. Every older #6840 plan is stale and must not be authorized.

[cv]

Please hold further sync-only pushes while the current validation runs. The last three main-merge commits have each invalidated an active CI/E2E evidence run, and GitHub can merge a clean behind branch safely. Please sync again only if a real conflict appears or a maintainer asks for it; we are trying to land this release target promptly.

[cjagwani]

Please hold further sync-only pushes while the current validation runs. The last three main-merge commits have each invalidated an active CI/E2E evidence run, and GitHub can merge a clean behind branch safely. Please sync again only if a real conflict appears or a maintainer asks for it; we are trying to land this release target promptly.

Done :)

[cjagwani]

@cv — the exact current-head E2E plan is ready for maintainer authorization. I have held further sync-only pushes as requested.

• PR: 6840
• Head: bb677130f1715894a278bb2b5c109299f844c5d9
• Base: a8c47d7dde4278c67209dc46566e448c32de1c55
• Deterministic plan: 6a916e97515d840f3fdaae0cd7b912e4ffa48db31eae1867bb0227b8da77203c
• Coordination check: https://github.com/NVIDIA/NemoClaw/runs/87433198565
• Selected jobs: cloud-onboard, credential-sanitization, security-posture, gpu-e2e, onboard-repair, onboard-resume, rebuild-hermes, rebuild-hermes-stale-base

The authoritative gate-producing ordinary CI run is green, the exact-head primary advisor has 0 blockers / 0 warnings / 0 suggestions, CodeRabbit has no unresolved thread, all commits are Verified and DCO-signed, and the PR is mergeable. No selected E2E job has run and no repository secret has been exposed.

Please run run-control-plane in the E2E / PR Gate Controller workflow with the PR/head/base above and review reason: Exact-head release stabilization validation, including Hermes rebuild image cleanup and stale-base recovery.

[cjagwani]

@cv — the fresh exact-diff E2E plan is ready after your signed empty retry commit.

• PR: 6840
• Head: 90663d2513d7cd16ee5a83197b474a47c3dcda69
• Base: a8c47d7dde4278c67209dc46566e448c32de1c55
• Deterministic plan: ae06de5ffbd430409feb74f5186eb442c6b2ca9f5659f980e1df88019382d54c
• Coordination check: https://github.com/NVIDIA/NemoClaw/runs/87442479478
• Selected jobs: cloud-onboard, credential-sanitization, security-posture, gpu-e2e, onboard-repair, onboard-resume, rebuild-hermes, rebuild-hermes-stale-base

The new-head authoritative CI is green, all eight CLI shards passed, the canonical primary advisor is high-confidence with 0 blockers / 0 warnings / 0 suggestions, CodeRabbit has no unresolved thread, and the PR remains approved and mergeable. No selected job has run for this exact SHA and no repository secret was exposed.

The prior bb677130… authorization is superseded. Its child run was canceled by the controller after one transient GitHub API HTTP 500 while polling status (not by a test failure): https://github.com/NVIDIA/NemoClaw/actions/runs/29439957771/job/87436076168

Please launch a fresh first-attempt run-control-plane with the exact PR/head/base above and review reason: Retry exact-head release stabilization validation after transient controller API outage.