fix(inference): extend probe retry budget for sustained 429 bursts

[hunglp6d]

Summary

✨ [AI-generated PR]

The nightly E2E run #27483389195 (2026-06-14, commit 136f656) failed across 17 of 19 failed jobs because the hosted inference endpoint (inference-api.nvidia.com/v1, model nvidia/nvidia/nemotron-3-super-v3) returned sustained HTTP 429 (Too Many Requests) during the concurrent probe window. The existing 3-retry / 50 s backoff budget (HTTP_PROBE_RETRY_DELAYS_MS = [5_000, 15_000, 30_000]) was exhausted before the API recovered, causing widespread endpoint validation failed exits and downstream sandbox-side inference.local failures.

This PR adds a 4th retry at 60 s (total budget: 110 s) so the probe can absorb longer rate-limiting windows without altering the fast path for healthy endpoints.

Changes

• src/lib/inference/onboard-probes.ts: extend HTTP_PROBE_RETRY_DELAYS_MS from [5_000, 15_000, 30_000] to [5_000, 15_000, 30_000, 60_000]
• Added a comment documenting the 110 s total budget rationale

Validation

Note: The custom-e2e validation branch could not be pushed because GITHUB_TOKEN lacks the workflow scope required to create workflow files on the upstream repo. A manual re-run of agent-turn-latency-e2e (or any of the 17 affected jobs) on this branch is recommended to confirm the fix absorbs the 429 burst.

• Original failing run: #27483389195 on 136f65662841ac16f1614e605dce14440458575b
• Targeted jobs: agent-turn-latency-e2e (#81234958152), openclaw-tui-chat-correlation-e2e (#81234958284), token-rotation-e2e (#81234958294), sandbox-operations-e2e (#81234958312), openclaw-slack-pairing-e2e (#81234958365), inference-routing-e2e (#81234958366), messaging-providers-e2e (#81234958372), sessions-agents-cli-e2e (#81234958373), common-egress-agent-e2e (#81234958405), channels-stop-start-openclaw-e2e (#81234958439), network-policy-e2e (#81234958458), openclaw-discord-pairing-e2e (#81234958475), upgrade-stale-sandbox-e2e (#81234958487), state-backup-restore-e2e (#81234958502), tunnel-lifecycle-e2e (#81234958504), rebuild-openclaw-e2e (#81234958518), rebuild-hermes-stale-base-e2e (#81234958528)

Type of Change

• Code change (feature, bug fix, or refactor)

Verification

• No secrets, API keys, or credentials committed
• npx prek run --all-files passes
• npm test passes

AI Disclosure

• AI-assisted — tool: Claude Code

---

Signed-off-by: Hung Le hple@nvidia.com

Fixes #5408

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 393328be-f7e8-4cac-9440-2c2d18b8a2f1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/nightly-e2e-inference-api-rate-limit-136f656

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the fix/nightly-e2e-infe... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.

File	main	fix/nightly-e2e-infe... afb42fb	+/-
nemoclaw/src/se...cret-scanner.ts	—	100%	—
nemoclaw/src/commands/slash.ts	—	100%	—
nemoclaw/src/li...bprocess-env.ts	—	100%	—
nemoclaw/src/bl...eprint/state.ts	—	98%	—
nemoclaw/src/onboard/config.ts	—	98%	—
nemoclaw/src/bl...int/snapshot.ts	—	97%	—
nemoclaw/src/bl...print/runner.ts	—	95%	—
nemoclaw/src/co...ration-state.ts	—	94%	—
nemoclaw/src/bl...ate-networks.ts	—	94%	—
nemoclaw/src/index.ts	—	94%	—

TypeScript / code-coverage/cli

The overall coverage in the fix/nightly-e2e-infe... branch is 44%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.

File	main	fix/nightly-e2e-infe... afb42fb	+/-
src/lib/state/o...oard-session.ts	—	90%	—
src/lib/inference/local.ts	—	77%	—
src/lib/sandbox/config.ts	—	72%	—
src/lib/inference/nim.ts	—	72%	—
src/lib/onboard/preflight.ts	—	64%	—
src/lib/state/sandbox.ts	—	55%	—
src/lib/onboard...er-gpu-patch.ts	—	50%	—
src/lib/actions...licy-channel.ts	—	49%	—
src/lib/policy/index.ts	—	48%	—
src/lib/onboard.ts	—	17%	—

---

_{Updated June 14, 2026 01:05 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.}

[github-actions]

E2E Advisor Recommendation

Required E2E: cloud-onboard-e2e
Optional E2E: cloud-inference-e2e, inference-routing-vitest

Dispatch hint: cloud-onboard-e2e

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

• cloud-onboard-e2e (medium): Runs the live public install/onboard flow with NVIDIA hosted inference credentials and exercises the validation probe path affected by the retry-budget change.

Optional E2E

• cloud-inference-e2e (medium): Adds confidence that a sandbox onboarded through the affected provider-validation path can still route a live inference.local chat request end-to-end.
• inference-routing-vitest (medium): PR-safe adjacent coverage for inference routing error classification and cleanup around invalid credentials/unreachable endpoints, useful because the touched code is in provider validation/routing setup.

New E2E recommendations

• onboarding validation (medium): Existing live E2E can exercise successful provider validation, but it does not deterministically prove that a sequence of HTTP 429/5xx probe responses is retried through the full backoff budget before success. A hermetic fake OpenAI-compatible endpoint would make this regression reproducible without relying on real provider rate limits.
  • Suggested test: Add a fake-endpoint onboard E2E scenario where the validation endpoint returns repeated HTTP 429 responses followed by success, with sleeps disabled/shortened, and assert onboarding succeeds after the expected retry count.

Dispatch hint

• Workflow: .github/workflows/nightly-e2e.yaml
• jobs input: cloud-onboard-e2e

[github-actions]

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: ubuntu-repo-cloud-openclaw
Optional Vitest E2E scenarios: None

Dispatch required Vitest E2E scenarios:

• gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required Vitest E2E scenarios

• ubuntu-repo-cloud-openclaw: The PR changes hosted inference onboarding validation probe retry behavior. The live-supported Ubuntu cloud OpenClaw scenario exercises NVIDIA cloud onboarding and the inference/credentials suites that depend on these validation probes.
  • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Optional Vitest E2E scenarios

• None.

Relevant changed files

• src/lib/inference/onboard-probes.ts

[github-actions]

PR Review Advisor

Findings: 0 needs attention, 2 worth checking, 0 nice ideas
Top item: Add coverage for the fourth retry budget

Review findings

🛠️ Needs attention

• None.

🔎 Worth checking

• Source-of-truth review needed: Hosted inference validation retry recovery: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: The changed comment cites sustained 429 bursts from concurrent nightly jobs, and the retry delay array now adds a 60s fourth delay.
• Fourth retry budget is not covered by a regression test (src/lib/inference/onboard-probes.ts:241): The existing retry tests only cover one HTTP 429 followed by success, so they would pass with both the old three-delay schedule and the new four-delay schedule. The changed constant is also reused by the timeout/connection-failure retry path, so the 60s extension affects more than sustained HTTP 429s. Without a focused mocked-boundary test, this recovery behavior can regress back to the old budget or expand unintentionally.
  • Recommendation: Add mocked `runCurlProbe`/`Atomics.wait` coverage that returns four HTTP 429 responses followed by success and asserts five probe calls plus waits of `[5000, 15000, 30000, 60000]`; also cover final failure after all four delays. If the timeout/connection-failure path should not inherit the new 60s delay, split the retry constants.
  • Evidence: `HTTP_PROBE_RETRY_DELAYS_MS` changes to `[5_000, 15_000, 30_000, 60_000]` at line 241; existing `test/wsl2-probe-timeout.test.ts` only asserts a single throttled 429 then success.

🌱 Nice ideas

• None.

Consider writing more tests for

• **Mocked behavioral coverage** — Mock `runCurlProbe` to return four HTTP 429 results followed by success and assert `probeOpenAiLikeEndpoint` succeeds after five probe calls with waits `[5000, 15000, 30000, 60000]`.. The changed code controls network/process retry behavior and is already testable with mocked `runCurlProbe` and `Atomics.wait`; existing coverage does not distinguish the new four-delay budget from the previous three-delay budget.
• **Mocked behavioral coverage** — Mock repeated HTTP 429 results and assert validation fails after exactly five probe calls rather than retrying indefinitely.. The changed code controls network/process retry behavior and is already testable with mocked `runCurlProbe` and `Atomics.wait`; existing coverage does not distinguish the new four-delay budget from the previous three-delay budget.
• **Mocked behavioral coverage** — If sharing the extended schedule is intentional, mock timeout/connection-failure retry exhaustion and assert the same bounded four-delay schedule applies; otherwise add a test proving timeout retries keep their intended shorter budget.. The changed code controls network/process retry behavior and is already testable with mocked `runCurlProbe` and `Atomics.wait`; existing coverage does not distinguish the new four-delay budget from the previous three-delay budget.
• **Fourth retry budget is not covered by a regression test** — Add mocked `runCurlProbe`/`Atomics.wait` coverage that returns four HTTP 429 responses followed by success and asserts five probe calls plus waits of `[5000, 15000, 30000, 60000]`; also cover final failure after all four delays. If the timeout/connection-failure path should not inherit the new 60s delay, split the retry constants.
• **Acceptance clause:** This PR adds a **4th retry at 60 s** (total budget: 110 s) so the probe can absorb longer rate-limiting windows without altering the fast path for healthy endpoints. — add test evidence or identify existing coverage. The fourth 60s delay is implemented and the initial probe still runs before any sleep, preserving the healthy fast path in code. However, no test demonstrates recovery after four consecutive 429s, and the shared constant also extends timeout/connection-failure retries.
• **Acceptance clause:** `npx prek run --all-files` passes — add test evidence or identify existing coverage. The PR body leaves this unchecked, and this review did not execute commands or evaluate external CI.
• **Acceptance clause:** `npm test` passes — add test evidence or identify existing coverage. The PR body leaves this unchecked, and this review did not execute commands or evaluate external CI.
• **Hosted inference validation retry recovery** — Existing tests cover a single HTTP 429 followed by success but do not prove the new fourth delay/fifth attempt or final bounded failure behavior.. The changed comment cites sustained 429 bursts from concurrent nightly jobs, and the retry delay array now adds a 60s fourth delay.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

[hunglp6d]

This is AI-generated PR.
Closed due to flaky