fix: let OpenShell routes converge

[ericksoa]

Summary

OpenShell 0.0.101 refreshes its sandbox route cache every 5 seconds, but the existing cross-family verification exhausted after 3 seconds.
This change keeps the same three-probe policy while moving the retries to 2 and 4 seconds, so the final sandbox probe occurs after one complete cache-refresh interval.

Related Issue

Fixes #9467

Changes

• Delay cross-family HTTP 400/404 sandbox route probes by 2 seconds and then 4 seconds.
• Log only the HTTP status, probe number, and delay for each scheduled retry.
• Keep authentication, server, transport, unsafe, malformed-input, and every other failure terminal.
• Assert exact exhaustion rollback, provider deletion, and absence of durable registry, config, and session writes.
• Record the bounded policy in the E2E retry inventory and update the provider-switch documentation.

Acceptance Criteria

• The trusted manual PR E2E target openclaw-inference-switch passes on the exact final PR head, reproducing the Inference: OpenClaw switches providers and remains responsive / no provider credential job that failed in run 32132319706. Its artifact records target openclaw-inference-switch as passed on exact head 93fd81965a65976578cfd5c51873100ce0de5828 in run 32162923306.
• The complete unfiltered trusted manual PR E2E matrix was rerun on the production-identical parent head before the final test-only deduplication commit: run 32152902405. The candidate-relevant OpenClaw provider-switch and advisor-selected Hermes provider-switch lanes passed; unrelated failures were classified as the shared hosted endpoint's HTTP 429 wave, an absent repository BRAVE_API_KEY, a current trusted Deep Agents base-image import failure, and a Telegram baseline assertion also reproduced in another PR's full matrix.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: reviewed the exact head against the security rubric; retries repeat only a read-only sandbox probe after one route mutation, the allowlist remains cross-family HTTP 400/404, progress logs contain no body, header, endpoint, or credential, HTTP 401 and 500 are regression-tested as terminal, and exhaustion restores the prior route and removes the uncommitted provider without durable writes.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit: Not applicable.
• Station profile/scenario: Not applicable.
• Result: Not applicable.
• Supporting evidence: Not applicable.

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run src/lib/actions/inference-set-compatible-provider.test.ts: 36 passed; npm run typecheck; npm run typecheck:cli -- --incremental; npm run lint; npm run docs.
• Applicable broad gate passed — not applicable; this changes one bounded retry schedule and its focused regression coverage, not the broad runtime, test harness, or coverage baseline.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Aaron Erickson aerickson@nvidia.com

Summary by CodeRabbit

• Bug Fixes

  • Improved provider switching after API-family changes by using more reliable retry timing.
  • Added clearer retry status and attempt information during sandbox route convergence.
  • Preserved immediate failure behavior for non-retryable HTTP errors.

• Documentation

  • Documented retry behavior, including delays, limits, and rollback handling.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 97125ad0-2ffa-4f03-ac0d-94b45e947875

📥 Commits

Reviewing files that changed from the base of the PR and between 097b734 and 93fd819.

📒 Files selected for processing (1)

• src/lib/actions/inference-set-compatible-provider.test.ts

Included review availability: Your plan includes up to 12 reviews per rolling hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

Sandbox inference route convergence now retries eligible HTTP 400/404 probe failures after 2 and 4 seconds. Retry logs include status and attempt details. Tests cover exhaustion, rollback, terminal failures, cleanup, and unchanged session state.

Changes

Sandbox route convergence

Layer / File(s)	Summary
Retry contract and logging src/lib/actions/inference-set-provider.ts, src/lib/actions/inference-set.ts, docs/inference/switch-providers.mdx, test/e2e/RETRY_INVENTORY.md	Route convergence uses 2-second and 4-second delays. An optional retry callback reports probe results, delays, and attempts. Retry logs and inventory documentation describe the behavior.
Convergence validation and rollback src/lib/actions/inference-set-compatible-provider.test.ts	Tests verify retry timing and logs, retry exhaustion, restoration of the previous provider and model, deletion of the temporary provider, unchanged session state, and non-retryable HTTP 401 and 500 failures.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 93fd8

This PR extends sandbox route verification delays so provider switching can converge without changing terminal failure handling or rollback behavior; no actionable merge-blocking risk remains.

Sequence Diagram(s)

sequenceDiagram
  participant inference_set
  participant probe_route
  participant retryUntilAsync
  participant sandbox
  participant session_state
  inference_set->>probe_route: validate target route
  probe_route->>sandbox: send inference probe
  sandbox-->>probe_route: return HTTP 400 or 404
  probe_route->>retryUntilAsync: wait 2s or 4s and retry
  retryUntilAsync->>probe_route: run next probe
  probe_route-->>inference_set: return convergence result
  inference_set->>session_state: persist route or restore prior selection

Loading

Suggested reviewers: prekshivyas, cv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes satisfy #9467 by adding bounded 400/404 retries, rollback, cleanup, non-retryable failure coverage, and durable-state protections.
Out of Scope Changes check	✅ Passed	The code, tests, documentation, and retry inventory changes directly support the linked issue and stated route-convergence objectives.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: enabling OpenShell route convergence during provider switching.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/issue-9467-openclaw-route-convergence

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-9481.docs.buildwithfern.com/nemoclaw

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 93fd819 in the fix/issue-9467-openc... branch remains at 96%, unchanged from commit d583fa1 in the main branch.

Show a code coverage summary of the most impacted files.

File	main d583fa1	fix/issue-9467-openc... 93fd819	+/-
nemoclaw/src/bl...ate-networks.ts	89%	94%	+5%

TypeScript / code-coverage/cli

The overall coverage in commit 93fd819 in the fix/issue-9467-openc... branch remains at 83%, unchanged from commit 9ab3cd3 in the main branch.

---

_{Updated August 18, 2026 17:58 UTC}

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized terminology decisions differ; normalized E2E selections differ; severity counts match.

3 terminology differences from the second opinion

Advisory only. These are normalized differences from the primary terminology receipt.

• inference invocation probe at src/lib/actions/inference-set-compatible-provider.test.ts:838: selected only by the second-opinion lane as established.
• API family at docs/inference/switch-providers.mdx:78: selected only by the second-opinion lane as established.
• route cache at src/lib/actions/inference-set-provider.ts:36: selected only by the second-opinion lane as define.

Second-opinion terminology and E2E selections are advisory. Live E2E does not run automatically for pull requests.

3 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

• established — route convergence at src/lib/actions/inference-set.ts:1177: Keep `route convergence`; existing repository usage has the same meaning.
• established — sandbox inference probe at test/e2e/RETRY_INVENTORY.md:22: Keep `sandbox inference probe`; existing tests use the same term.
• define — uncommitted provider at test/e2e/RETRY_INVENTORY.md:22: Use `newly created provider` for consistency with the provider-switch documentation, or define `uncommitted provider` in the inventory row.

E2E guidance

Advisory only. A maintainer can dispatch the default E2E suite for the commit under review.

Recommended E2E: inference-routing

Manual-only E2E: network-policy
The manual PR workflow does not run these selectors for the commit under review. Run them from reviewed code on main.

2 optional E2E recommendations

• openclaw-inference-switch
• hermes-inference-switch

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[jyaunches]

LOC Reduction / Codebase Simplicity Review

Why this blocks

This PR is +123/-6, and 102 of those net new lines are in src/lib/actions/inference-set-compatible-provider.test.ts. The new does not retry a changed-family ... failure table adds about 99 lines for two status values, but it repeats the adjacent changed-family exhaustion case’s complete createCompatibleProviderCapture setup, createDeps state, runInferenceSet call, two-command rollback history, provider deletion check, and no-durable-write assertions. The existing does not retry a target rejection when the API family did not change and restores the prior route when sandbox-only provider verification ... cases repeat the same scenario again.

The only new dimensions are HTTP 401/500, one probe, and no sleep. Those do not require another full switch fixture.

Refactor direction

Make one rejected-compatible-switch scenario helper, or parameterize one table by API-family change, HTTP status, expected probe count, and expected delays. Have the existing same-family 400 case, the changed-family 401/500 cases, and the exhaustion case reuse the same setup and rollback/no-write assertions. Keep only the expectations that differ per row.

Expected result

The terminal-status coverage and exact rollback proof remain, but the new coverage becomes a small table rather than a copied 99-line workflow. This should remove roughly 60–80 lines and make the retry policy easier to audit as one matrix.

[ericksoa]

Addressed the LOC/simplicity review in 93fd81965a65976578cfd5c51873100ce0de5828: the same-family rejection, changed-family terminal failures, and retry-exhaustion cases now share one rejected-compatible-switch scenario helper while retaining the exact rollback and no-durable-write assertions. The test file change is 143 additions / 218 deletions (net -75 lines). The 36 targeted tests, both TypeScript checks, Oxlint, repository checks, and commit hooks pass. @jyaunches please re-review when convenient.

Resolved at 93fd819. The rejected compatible-switch cases now share one scenario owner, preserving rollback and no-write assertions while removing 75 net test lines.

[jyaunches]

LOC Reduction / Codebase Simplicity Review

Resolved at 93fd81965a65976578cfd5c51873100ce0de5828.

runRejectedCompatibleSwitchScenario now owns the compatible-provider setup, switch, rollback command history, provider deletion, no-durable-write assertions, and restored session state. The exhausted-retry, terminal 401/500, and same-family rejection cases retain only their differing probe, delay, and log expectations.

The follow-up removes 75 net test lines from the reviewed commit while preserving the retry-policy matrix. I found no replacement LOC-reduction or codebase-simplicity blocker in the current complete diff. This closes only the prior simplicity review; it is not an approval of other review dimensions.

[ericksoa]

Advisor follow-up for the partial Nemotron attempt:

• 5-second cache interval / runtime evidence: OpenShell v0.0.101 defines DEFAULT_ROUTE_REFRESH_INTERVAL_SECS = 5 in inference_routes.rs, and NemoClaw pins both its minimum and maximum supported OpenShell versions to 0.0.101 in nemoclaw-blueprint/blueprint.yaml. The exact E2E job that failed in OpenClaw provider-switch E2E still returns HTTP 400 after route convergence fix #9467 now passes on final head 93fd81965a65976578cfd5c51873100ce0de5828: run 32162923306 / job 95796719911. Its manifest binds that SHA and target, and target-result.json records openclaw-inference-switch: passed.
• Idempotence basis: the Idempotence basis cell already says each retry repeats only the read-only sandbox inference probe after the single route mutation; the emitted E2E retry evidence additionally records idempotence: idempotent. No mutation is repeated.
• Overlap findings: this PR is currently MERGEABLE. I am sequencing fix: let OpenShell routes converge #9481 before the still-open feat(cli): add Hermes portable lifecycle authority #9424 and fix(e2e): bind PR runs to managed image cohort #9484; they can refresh after this lands. No merge/rebase of main is needed here.
• The two other suggestion-shaped entries explicitly say the docs match code and state ordering preserves invariants; they require no change.

No source change is needed for these findings: the requested authoritative version evidence, exact live E2E, idempotence basis, and merge sequencing are now recorded.