fix(onboard): retry transient inference smoke - #5932
Merged
Conversation
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Contributor
📝 WalkthroughWalkthroughAdds configurable retry parameters to the compatible-endpoint sandbox smoke script, changes the generated shell flow to loop over bounded attempts, updates timeout and non-JSON diagnostics, and adjusts unit and e2e tests to cover the retry behavior. Smoke Script Retry Loop
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Contributor
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in the Show a code coverage summary of the most covered files.
TypeScript / code-coverage/cliThe overall coverage in the Show a code coverage summary of the most covered files.
Updated |
Contributor
E2E Advisor RecommendationRequired E2E: Dispatch hint: Auto-dispatched E2E: Full advisor summaryE2E Recommendation AdvisorBase: Required E2E
Optional E2E
New E2E recommendations
Dispatch hint
|
Contributor
Vitest E2E Scenario RecommendationRequired Vitest E2E scenarios: Dispatch required Vitest E2E scenarios:
Full Vitest E2E advisor summaryVitest E2E Scenario AdvisorBase: Required Vitest E2E scenarios
Optional Vitest E2E scenarios
Relevant changed files
|
Contributor
PR Review Advisor — Changes requestedMerge posture: Do not merge yet Action checklist
Findings index
Review findings by urgency: 0 required fixes, 2 items to resolve/justify, 0 in-scope improvements
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/onboard/compatible-endpoint-smoke.ts`:
- Around line 349-360: The smoke test retry flow in compatible-endpoint-smoke
should distinguish transient non-JSON gateway responses from terminal validation
failures instead of treating every non-zero from check_response the same. Update
check_response to return a distinct retryable status only when can_retry is true
and the body is non-JSON, and ensure the outer attempt loop around
check_response/$status exits immediately for permanent JSON-shape/content
validation failures rather than sleeping and retrying. Use the existing
check_response, can_retry, and status handling in compatible-endpoint-smoke to
keep the retry loop limited to genuinely transient failures.
In `@test/e2e/test-hermes-e2e.sh`:
- Around line 625-638: The direct API retry path is losing the useful raw
gateway response when parse_chat_content fails, so the exhausted failure message
can end up empty. In the direct API loop, keep api_response as the fallback
source and use it in the final fail branch when api_content is empty or parsing
fails, so the retry exhaustion log still shows the raw 504/body text. Update the
logic around parse_chat_content, api_content, and the final pass/fail checks to
preserve that payload for debugging.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8364524c-0c41-4083-ad87-6debb404f7a1
📒 Files selected for processing (3)
src/lib/onboard/compatible-endpoint-smoke.test.tssrc/lib/onboard/compatible-endpoint-smoke.tstest/e2e/test-hermes-e2e.sh
Contributor
PR Review Advisor (Nemotron Ultra) — Changes requestedMerge posture: Do not merge yet Action checklist
Findings index
🚨 Required before mergeAddress these before merging unless a maintainer explicitly overrides the advisor with rationale.
|
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Collaborator
Author
|
Advisor disposition for commit 4c5abf1:
Current-head token-rotation and Hermes runtime validations are being run sequentially to avoid hosted-service contention. |
Contributor
Selective E2E Results — ✅ All requested jobs passedRun: 28326589749
|
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Contributor
There was a problem hiding this comment.
♻️ Duplicate comments (1)
src/lib/onboard/compatible-endpoint-smoke.ts (1)
364-381: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winTerminal validation failures are still retried with backoff.
check_responseexits1for transient non-JSON gateway bodies and for deterministic JSON-shape/content failures (Lines 341, 347), and the loop retries any non-zero status until the budget is exhausted. A permanently bad-but-valid JSON response therefore issues extra smoke requests and sleeps on backoff instead of failing fast. Consider a distinct retryable code for the transient non-JSON case and exiting immediately for non-retryable validation failures.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/onboard/compatible-endpoint-smoke.ts` around lines 364 - 381, The smoke loop in compatible-endpoint-smoke.ts treats every non-zero result from check_response as retryable, so deterministic JSON validation failures get retried with backoff. Update check_response to distinguish transient non-JSON gateway bodies from permanent JSON-shape/content failures using a distinct exit status, then adjust the surrounding retry logic in the run_smoke_request/attempt loop to retry only the transient case and fail fast on non-retryable validation errors.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@src/lib/onboard/compatible-endpoint-smoke.ts`:
- Around line 364-381: The smoke loop in compatible-endpoint-smoke.ts treats
every non-zero result from check_response as retryable, so deterministic JSON
validation failures get retried with backoff. Update check_response to
distinguish transient non-JSON gateway bodies from permanent JSON-shape/content
failures using a distinct exit status, then adjust the surrounding retry logic
in the run_smoke_request/attempt loop to retry only the transient case and fail
fast on non-retryable validation errors.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f0e8ede4-fe57-4849-9f2d-246c2ebeaed2
📒 Files selected for processing (2)
src/lib/onboard/compatible-endpoint-smoke.test.tssrc/lib/onboard/compatible-endpoint-smoke.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- src/lib/onboard/compatible-endpoint-smoke.test.ts
Contributor
Selective E2E Results —
|
| Job | Result |
|---|---|
| messaging-compatible-endpoint-e2e |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e/test-hermes-e2e.sh`:
- Around line 632-644: The live API probe in the test script is matching on any
parsed text containing PONG, which can let non-exact responses slip through.
Update the checks around parse_chat_content and the final pass in the direct API
loop to compare the normalized api_content against PONG exactly, so only an
exact one-word response breaks and passes the test.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 65c97f27-ff2d-4a08-8f9e-41661eb47f9d
📒 Files selected for processing (3)
src/lib/onboard/compatible-endpoint-smoke.test.tssrc/lib/onboard/compatible-endpoint-smoke.tstest/e2e/test-hermes-e2e.sh
Comment on lines
+632
to
+644
| api_content="$(printf '%s' "$api_response" | parse_chat_content 2>/dev/null || true)" | ||
| api_failure_summary="response without PONG (HTTP ${api_http_status: -3}, chars=${#api_response})" | ||
| if grep -qi "PONG" <<<"$api_content"; then | ||
| pass "[LIVE] Direct API: model responded with PONG" | ||
| else | ||
| fail "[LIVE] Direct API: expected PONG, got: ${api_content:0:200}" | ||
| break | ||
| fi | ||
| if [ "$attempt" -lt 3 ]; then | ||
| info "[LIVE] Direct API attempt ${attempt}/3 ${api_failure_summary}; retrying..." | ||
| sleep $((5 * attempt)) | ||
| fi | ||
| done | ||
|
|
||
| if grep -qi "PONG" <<<"$api_content"; then | ||
| pass "[LIVE] Direct API: model responded with PONG" |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Require an exact PONG match here.
Lines 634 and 643 currently pass on any parsed text that merely contains PONG, so outputs like NOT PONG or extra explanatory text can satisfy the probe without proving the advertised one-word response. Compare the normalized parsed content to PONG exactly before breaking/passing.
As per path instructions, tests should “prefer observable outcomes through the public boundary” and flag “conditionals that make a test pass without exercising its claim.”
Suggested change
- if grep -qi "PONG" <<<"$api_content"; then
+ if printf '%s\n' "$api_content" | grep -Eqix 'PONG'; then
break
fi
@@
-if grep -qi "PONG" <<<"$api_content"; then
+if printf '%s\n' "$api_content" | grep -Eqix 'PONG'; then
pass "[LIVE] Direct API: model responded with PONG"
else
fail "[LIVE] Direct API: expected PONG after 3 attempts; ${api_failure_summary}"
fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| api_content="$(printf '%s' "$api_response" | parse_chat_content 2>/dev/null || true)" | |
| api_failure_summary="response without PONG (HTTP ${api_http_status: -3}, chars=${#api_response})" | |
| if grep -qi "PONG" <<<"$api_content"; then | |
| pass "[LIVE] Direct API: model responded with PONG" | |
| else | |
| fail "[LIVE] Direct API: expected PONG, got: ${api_content:0:200}" | |
| break | |
| fi | |
| if [ "$attempt" -lt 3 ]; then | |
| info "[LIVE] Direct API attempt ${attempt}/3 ${api_failure_summary}; retrying..." | |
| sleep $((5 * attempt)) | |
| fi | |
| done | |
| if grep -qi "PONG" <<<"$api_content"; then | |
| pass "[LIVE] Direct API: model responded with PONG" | |
| api_content="$(printf '%s' "$api_response" | parse_chat_content 2>/dev/null || true)" | |
| api_failure_summary="response without PONG (HTTP ${api_http_status: -3}, chars=${`#api_response`})" | |
| if printf '%s\n' "$api_content" | grep -Eqix 'PONG'; then | |
| break | |
| fi | |
| if [ "$attempt" -lt 3 ]; then | |
| info "[LIVE] Direct API attempt ${attempt}/3 ${api_failure_summary}; retrying..." | |
| sleep $((5 * attempt)) | |
| fi | |
| done | |
| if printf '%s\n' "$api_content" | grep -Eqix 'PONG'; then | |
| pass "[LIVE] Direct API: model responded with PONG" | |
| else | |
| fail "[LIVE] Direct API: expected PONG after 3 attempts; ${api_failure_summary}" | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test/e2e/test-hermes-e2e.sh` around lines 632 - 644, The live API probe in
the test script is matching on any parsed text containing PONG, which can let
non-exact responses slip through. Update the checks around parse_chat_content
and the final pass in the direct API loop to compare the normalized api_content
against PONG exactly, so only an exact one-word response breaks and passes the
test.
Source: Path instructions
Contributor
Selective E2E Results — ✅ All requested jobs passedRun: 28326902209
|
21 tasks
cv
added a commit
that referenced
this pull request
Jun 28, 2026
<!-- markdownlint-disable MD041 --> ## Summary Tightens the compatible-endpoint smoke retry merged in #5932 so only explicit transient transport and HTTP signals retry. It also removes curl stderr/body dependence, derives exponential-backoff timeout coverage, and adds behavioral tests for transient and permanent curl exits. ## Changes - Capture HTTP status directly from curl instead of inferring it from HTML response bodies. - Suppress curl stderr and retry only transient exits 6, 7, 28, 52, 55, and 56. - Keep permanent curl and JSON validation failures single-shot and strict. - Align retry delay with the existing Vitest 5s/10s backoff and derive the 225-second parent timeout from the shared budget. - Centralize the generated retry policy and triangular timeout budget in a focused classifier module. - Add behavioral coverage for transient curl exits 6, 7, 28, 52, 55, and 56; permanent curl and HTTP 429 failures; hostile model quoting; and 5s/10s backoff. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check all that apply. For any "covered by existing tests", "not applicable", or waiver entry, add a brief justification on the same line or in the Changes section. --> - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: internal validation retry classification and diagnostics only; no user-facing configuration changes. - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: arbitrary curl stderr and upstream response bodies are suppressed; diagnostics retain only exit code, HTTP status, and response length, and all permanent validation failures remain strict. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification <!-- Check each item you ran and confirmed. Leave unchecked items you skipped. Doc-only changes do not require npm test unless you ran it. --> - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Git hooks passed during commit and push, or `npx prek run --from-ref main --to-ref HEAD` passes - [x] Targeted tests pass for changed behavior - [ ] Full `npm test` passes (broad runtime changes only) - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) Targeted verification: - `npx vitest run --project cli src/lib/onboard/compatible-endpoint-smoke.test.ts src/lib/onboard/smoke-retry-classifier.test.ts` (36 tests passed) - `npx biome check src/lib/onboard/compatible-endpoint-smoke.ts src/lib/onboard/compatible-endpoint-smoke.test.ts` - `npm run typecheck:cli` - `git diff --check` - Normal commit and push hooks passed, including the full CLI suite, repository checks, and secret scanning. - Token rotation reused-sandbox runtime: https://github.com/NVIDIA/NemoClaw/actions/runs/28327364831 (success). - Hermes dashboard runtime: https://github.com/NVIDIA/NemoClaw/actions/runs/28328424431 (success). Post-merge review findings addressed from #5932: - Curl-exit retry acceptance coverage: transient exits 6, 7, 28, 52, 55, and 56 recover on attempt two; permanent exit 2 makes exactly one request. - Curl stderr security: stderr is suppressed at the external boundary and only the numeric exit is reported. - Retry classification: only explicit transient transport codes, HTTP 5xx, and reasoning-budget exhaustion retry; HTTP 429 remains single-shot. - HTTP classification: status comes from curl metadata rather than fragile HTML matching. - Retry timing: 5s/10s backoff and its 225-second parent budget are derived from shared constants. --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved compatible-endpoint smoke verification with more deterministic HTTP-status handling to classify retries vs terminal failures, including clearer retry timing and faster exits for non-retryable outcomes. * **Tests** * Expanded smoke-test coverage for retry behavior: * Transient HTTP 5xx responses now retry and succeed. * HTTP 429 responses no longer retry. * Added scenarios for multiple transient curl exit codes (retries succeed) and terminal exit codes (fails immediately). <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
This was referenced Jun 28, 2026
21 tasks
cv
pushed a commit
that referenced
this pull request
Jun 29, 2026
## Summary Adds the v0.0.69 release notes to the published release-notes page so users can see the shipped sandbox recovery, Deep Agents Code, Hermes, inference, policy, and release-validation changes. The section is based on the v0.0.69 announcement and links each user-facing theme to the deeper docs pages that already cover the behavior. ## Changes - Added a new `v0.0.69` section to `docs/about/release-notes.mdx`. - Linked release-note themes to lifecycle, backup, troubleshooting, Deep Agents Code, commands, workspace, messaging, Hermes, inference, security, monitoring, and network-policy docs. Source summary: - #5455 -> `docs/about/release-notes.mdx`: Summarized persistent workspace and state cleanup during sandbox destroy. - #5738 -> `docs/about/release-notes.mdx`: Summarized nonzero exit status preservation for failed hosted endpoint validation. - #5786 -> `docs/about/release-notes.mdx`: Summarized live sandbox rediscovery when local registry state is missing. - #5881 -> `docs/about/release-notes.mdx`: Summarized the `nemo-deepagents` alias command surface. - #5594 -> `docs/about/release-notes.mdx`: Summarized the Hermes Agent 2026.6.19 update. - #5777 -> `docs/about/release-notes.mdx`: Summarized manifest-derived messaging channel support. - #5825 -> `docs/about/release-notes.mdx`: Summarized DeepSeek V4 Flash managed-vLLM defaults for DGX Station. - #5877 -> `docs/about/release-notes.mdx`: Summarized provider switch metadata preservation. - #5932 -> `docs/about/release-notes.mdx`: Summarized transient inference smoke retry behavior. - #5934 -> `docs/about/release-notes.mdx`: Summarized constrained inference smoke retry boundaries. - #5681 -> `docs/about/release-notes.mdx`: Summarized Shields config-hash sealing during auto-restore. - #5682 -> `docs/about/release-notes.mdx`: Summarized sandbox connect process-limit enforcement. - #5683 -> `docs/about/release-notes.mdx`: Summarized JSON agent failure provenance warnings. - #5711 -> `docs/about/release-notes.mdx`: Summarized sparse-source log breadcrumbs. - #5838 -> `docs/about/release-notes.mdx`: Summarized host-authoritative Shields status. - #5880 -> `docs/about/release-notes.mdx`: Summarized policy round-trip documentation updates. - #5886 -> `docs/about/release-notes.mdx`: Summarized network request approval-flow documentation updates. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [x] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [ ] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [x] Tests not applicable — justification: doc-only release-notes prose; no runtime behavior changed. - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Git hooks passed during commit and push, or `npx prek run --from-ref main --to-ref HEAD` passes - [ ] Targeted tests pass for changed behavior - [ ] Full `npm test` passes (broad runtime changes only) - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) `npm run docs` passed with 0 errors and the existing Fern light-mode accent contrast warning. `fern check --warnings` reported the same accent-color warning. --- Signed-off-by: Miyoung Choi <miyoungc@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added release notes for **v0.0.69**, covering improved sandbox lifecycle recovery (state preservation across destroy/recreate/rebuild/recovery/validation failures), clearer Deep Agents Code terminal/CLI behavior, and safer Hermes messaging/provider switching with manifest-driven channels. * Improved inference setup validation guidance, including handling of local/compatible endpoints and redaction of sensitive validation errors. * Refreshed release-gate documentation with clearer approval examples and validation behavior for NVIDIA API keys vs hosted inference keys. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Hadar301
pushed a commit
to Hadar301/NemoClaw-OpenShift
that referenced
this pull request
Jul 12, 2026
<!-- markdownlint-disable MD041 --> ## Summary Retries transient compatible-endpoint sandbox smoke failures during onboarding and aligns the retained Hermes direct hosted probe with its already-retrying Vitest counterpart. This prevents short provider/proxy propagation windows and empty hosted responses from masking the behavior under test while retaining a strict, bounded failure budget. ## Changes - Retry the production compatible-endpoint sandbox smoke up to three times with bounded backoff, including curl failures and non-JSON gateway responses. - Preserve the larger-token retry used for reasoning-only responses and fail after the configured attempt budget. - Add executable recovery and exhaustion tests using a fake `504 Gateway Time-out` response. - Retry the retained Hermes direct hosted `PONG` probe three times, matching the live Vitest lane. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check all that apply. For any "covered by existing tests", "not applicable", or waiver entry, add a brief justification on the same line or in the Changes section. --> - [x] Tests added or updated for changed behavior - [x] Existing tests cover changed behavior — justification: the live Hermes Vitest lane already exercises three-attempt direct and sandbox hosted inference retries. - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: bounded resilience for existing validation probes; no user-facing configuration or workflow changes. - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: retry scope is limited to the existing authenticated validation request; provider configuration, credential binding, redaction, and strict final failure behavior are unchanged. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification <!-- Check each item you ran and confirmed. Leave unchecked items you skipped. Doc-only changes do not require npm test unless you ran it. --> - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Git hooks passed during commit and push, or `npx prek run --from-ref main --to-ref HEAD` passes - [x] Targeted tests pass for changed behavior - [ ] Full `npm test` passes (broad runtime changes only) - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) Targeted verification: - `npx vitest run --project cli src/lib/onboard/compatible-endpoint-smoke.test.ts` (8 tests passed) - `npx biome check src/lib/onboard/compatible-endpoint-smoke.ts src/lib/onboard/compatible-endpoint-smoke.test.ts` - `bash -n test/e2e/test-hermes-e2e.sh` - `npx prek run shellcheck --files test/e2e/test-hermes-e2e.sh` - `npm run typecheck:cli` - `git diff --check` Runtime evidence on `c4ad10eceb811a8803700237a86978ac93142370`: - Full Nightly run 28324129704: 66 successful, 4 skipped, and two failures. Hermes dashboard had one empty direct hosted response while all dashboard and routed checks passed. Token rotation had one same-token reuse smoke receive an HTML 504 while all rotation assertions passed. - Isolated Hermes dashboard run 28325346077 passed on the same SHA. - Isolated token rotation run 28325550172 reproduced HTML 504 responses in same-token reuse phases 5 and 7 while all rebuild/rotation phases passed, confirming the bounded sandbox-smoke retry is required independently of full-gate load. --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable retry behavior for compatible-endpoint smoke checks (custom attempts and retry delay), including improved shell-script environment parameters. * **Bug Fixes** * Made smoke checks more resilient to transient non-JSON gateway errors and updated retry decisioning for 5xx conditions and max-token scenarios. * Updated sandbox smoke command timeout calculation to match retry and request settings. * Improved hosted inference endpoint e2e test retries with clearer “PONG not found” failure output. * **Tests** * Expanded sandbox smoke expectations to validate retry counts, timeouts, and curl invocation across success and multiple failure modes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Hadar301
pushed a commit
to Hadar301/NemoClaw-OpenShift
that referenced
this pull request
Jul 12, 2026
<!-- markdownlint-disable MD041 --> ## Summary Tightens the compatible-endpoint smoke retry merged in NVIDIA#5932 so only explicit transient transport and HTTP signals retry. It also removes curl stderr/body dependence, derives exponential-backoff timeout coverage, and adds behavioral tests for transient and permanent curl exits. ## Changes - Capture HTTP status directly from curl instead of inferring it from HTML response bodies. - Suppress curl stderr and retry only transient exits 6, 7, 28, 52, 55, and 56. - Keep permanent curl and JSON validation failures single-shot and strict. - Align retry delay with the existing Vitest 5s/10s backoff and derive the 225-second parent timeout from the shared budget. - Centralize the generated retry policy and triangular timeout budget in a focused classifier module. - Add behavioral coverage for transient curl exits 6, 7, 28, 52, 55, and 56; permanent curl and HTTP 429 failures; hostile model quoting; and 5s/10s backoff. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check all that apply. For any "covered by existing tests", "not applicable", or waiver entry, add a brief justification on the same line or in the Changes section. --> - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: internal validation retry classification and diagnostics only; no user-facing configuration changes. - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: arbitrary curl stderr and upstream response bodies are suppressed; diagnostics retain only exit code, HTTP status, and response length, and all permanent validation failures remain strict. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification <!-- Check each item you ran and confirmed. Leave unchecked items you skipped. Doc-only changes do not require npm test unless you ran it. --> - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Git hooks passed during commit and push, or `npx prek run --from-ref main --to-ref HEAD` passes - [x] Targeted tests pass for changed behavior - [ ] Full `npm test` passes (broad runtime changes only) - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) Targeted verification: - `npx vitest run --project cli src/lib/onboard/compatible-endpoint-smoke.test.ts src/lib/onboard/smoke-retry-classifier.test.ts` (36 tests passed) - `npx biome check src/lib/onboard/compatible-endpoint-smoke.ts src/lib/onboard/compatible-endpoint-smoke.test.ts` - `npm run typecheck:cli` - `git diff --check` - Normal commit and push hooks passed, including the full CLI suite, repository checks, and secret scanning. - Token rotation reused-sandbox runtime: https://github.com/NVIDIA/NemoClaw/actions/runs/28327364831 (success). - Hermes dashboard runtime: https://github.com/NVIDIA/NemoClaw/actions/runs/28328424431 (success). Post-merge review findings addressed from NVIDIA#5932: - Curl-exit retry acceptance coverage: transient exits 6, 7, 28, 52, 55, and 56 recover on attempt two; permanent exit 2 makes exactly one request. - Curl stderr security: stderr is suppressed at the external boundary and only the numeric exit is reported. - Retry classification: only explicit transient transport codes, HTTP 5xx, and reasoning-budget exhaustion retry; HTTP 429 remains single-shot. - HTTP classification: status comes from curl metadata rather than fragile HTML matching. - Retry timing: 5s/10s backoff and its 225-second parent budget are derived from shared constants. --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved compatible-endpoint smoke verification with more deterministic HTTP-status handling to classify retries vs terminal failures, including clearer retry timing and faster exits for non-retryable outcomes. * **Tests** * Expanded smoke-test coverage for retry behavior: * Transient HTTP 5xx responses now retry and succeed. * HTTP 429 responses no longer retry. * Added scenarios for multiple transient curl exit codes (retries succeed) and terminal exit codes (fails immediately). <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Hadar301
pushed a commit
to Hadar301/NemoClaw-OpenShift
that referenced
this pull request
Jul 12, 2026
## Summary Adds the v0.0.69 release notes to the published release-notes page so users can see the shipped sandbox recovery, Deep Agents Code, Hermes, inference, policy, and release-validation changes. The section is based on the v0.0.69 announcement and links each user-facing theme to the deeper docs pages that already cover the behavior. ## Changes - Added a new `v0.0.69` section to `docs/about/release-notes.mdx`. - Linked release-note themes to lifecycle, backup, troubleshooting, Deep Agents Code, commands, workspace, messaging, Hermes, inference, security, monitoring, and network-policy docs. Source summary: - NVIDIA#5455 -> `docs/about/release-notes.mdx`: Summarized persistent workspace and state cleanup during sandbox destroy. - NVIDIA#5738 -> `docs/about/release-notes.mdx`: Summarized nonzero exit status preservation for failed hosted endpoint validation. - NVIDIA#5786 -> `docs/about/release-notes.mdx`: Summarized live sandbox rediscovery when local registry state is missing. - NVIDIA#5881 -> `docs/about/release-notes.mdx`: Summarized the `nemo-deepagents` alias command surface. - NVIDIA#5594 -> `docs/about/release-notes.mdx`: Summarized the Hermes Agent 2026.6.19 update. - NVIDIA#5777 -> `docs/about/release-notes.mdx`: Summarized manifest-derived messaging channel support. - NVIDIA#5825 -> `docs/about/release-notes.mdx`: Summarized DeepSeek V4 Flash managed-vLLM defaults for DGX Station. - NVIDIA#5877 -> `docs/about/release-notes.mdx`: Summarized provider switch metadata preservation. - NVIDIA#5932 -> `docs/about/release-notes.mdx`: Summarized transient inference smoke retry behavior. - NVIDIA#5934 -> `docs/about/release-notes.mdx`: Summarized constrained inference smoke retry boundaries. - NVIDIA#5681 -> `docs/about/release-notes.mdx`: Summarized Shields config-hash sealing during auto-restore. - NVIDIA#5682 -> `docs/about/release-notes.mdx`: Summarized sandbox connect process-limit enforcement. - NVIDIA#5683 -> `docs/about/release-notes.mdx`: Summarized JSON agent failure provenance warnings. - NVIDIA#5711 -> `docs/about/release-notes.mdx`: Summarized sparse-source log breadcrumbs. - NVIDIA#5838 -> `docs/about/release-notes.mdx`: Summarized host-authoritative Shields status. - NVIDIA#5880 -> `docs/about/release-notes.mdx`: Summarized policy round-trip documentation updates. - NVIDIA#5886 -> `docs/about/release-notes.mdx`: Summarized network request approval-flow documentation updates. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [x] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [ ] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [x] Tests not applicable — justification: doc-only release-notes prose; no runtime behavior changed. - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Git hooks passed during commit and push, or `npx prek run --from-ref main --to-ref HEAD` passes - [ ] Targeted tests pass for changed behavior - [ ] Full `npm test` passes (broad runtime changes only) - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) `npm run docs` passed with 0 errors and the existing Fern light-mode accent contrast warning. `fern check --warnings` reported the same accent-color warning. --- Signed-off-by: Miyoung Choi <miyoungc@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added release notes for **v0.0.69**, covering improved sandbox lifecycle recovery (state preservation across destroy/recreate/rebuild/recovery/validation failures), clearer Deep Agents Code terminal/CLI behavior, and safer Hermes messaging/provider switching with manifest-driven channels. * Improved inference setup validation guidance, including handling of local/compatible endpoints and redaction of sensitive validation errors. * Refreshed release-gate documentation with clearer approval examples and validation behavior for NVIDIA API keys vs hosted inference keys. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
wscurran
added
area: inference
wscurran
added
bug-fix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Retries transient compatible-endpoint sandbox smoke failures during onboarding and aligns the retained Hermes direct hosted probe with its already-retrying Vitest counterpart. This prevents short provider/proxy propagation windows and empty hosted responses from masking the behavior under test while retaining a strict, bounded failure budget.
Changes
504 Gateway Time-outresponse.PONGprobe three times, matching the live Vitest lane.Type of Change
Quality Gates
Verification
Verifiedin GitHubnpx prek run --from-ref main --to-ref HEADpassesnpm testpasses (broad runtime changes only)npm run docsbuilds without warnings (doc changes only)Targeted verification:
npx vitest run --project cli src/lib/onboard/compatible-endpoint-smoke.test.ts(8 tests passed)npx biome check src/lib/onboard/compatible-endpoint-smoke.ts src/lib/onboard/compatible-endpoint-smoke.test.tsbash -n test/e2e/test-hermes-e2e.shnpx prek run shellcheck --files test/e2e/test-hermes-e2e.shnpm run typecheck:cligit diff --checkRuntime evidence on
c4ad10eceb811a8803700237a86978ac93142370:Signed-off-by: Carlos Villela cvillela@nvidia.com
Summary by CodeRabbit