fix(onboard): summarize inference validation failures

[cv]

Summary

Improves non-interactive onboarding diagnostics for hosted/custom inference validation failures by printing a credential-safe probe summary. Operators can now distinguish HTTP statuses such as 404/429 from curl/timeout failures without exposing raw response bodies or credentials.

Related Issue

Related to nightly run 27504927177.

Changes

• Add src/lib/onboard/probe-diagnostics.ts to summarize probe failures as HTTP status, curl exit, or timeout-only diagnostics.
• Update inference selection validation to print Validation probe summary: ... before the redaction notice.
• Add tests proving raw provider bodies, stderr, and credential-shaped values are not surfaced in summaries.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Verification

• Git hooks passed during commit and push, or npx prek run --from-ref main --to-ref HEAD passes
• Targeted tests pass for changed behavior
• Full npm test passes (broad runtime changes only)
• Tests added or updated for new or changed behavior
• No secrets, API keys, or credentials committed
• Docs updated for user-facing behavior changes
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

Targeted verification:

• npx biome check --write src/lib/onboard/inference-selection-validation.ts src/lib/onboard/probe-diagnostics.ts src/lib/onboard/probe-diagnostics.test.ts
• npx vitest run --project cli src/lib/onboard/probe-diagnostics.test.ts
• npm run typecheck:cli

Docs review: no user-facing docs changes needed; this changes CLI diagnostic wording only.

---

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

• New Features

  • Added user-friendly probe failure diagnostics, including a formatted summary for validation failures when probe details are available.

• Bug Fixes

  • Improved validation error reporting by surfacing clearer HTTP/curl/timeout context while omitting raw response/error details that could expose sensitive information.

• Tests

  • Added a Vitest suite covering probe diagnostic summarization, including HTTP failures, curl/timeout failures, and fallback behavior when structured details are incomplete.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8a690b83-a208-4aac-b762-f6b711f9cac7

📥 Commits

Reviewing files that changed from the base of the PR and between 17765d1 and f002451.

📒 Files selected for processing (1)

• src/lib/onboard/inference-selection-validation.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• src/lib/onboard/inference-selection-validation.ts

---

📝 Walkthrough

Walkthrough

Adds a new probe-diagnostics.ts module exporting summarizeProbeForDisplay, which converts raw probe failure data into a redacted human-readable string by inspecting structured failures or parsing the message field. The printValidationFailure helper in inference-selection-validation.ts is extended to accept and log probe summaries, and four validation call sites now pass the failing probe.

Changes

Probe Diagnostics and Validation Logging

Layer / File(s)	Summary
summarizeProbeForDisplay implementation and tests src/lib/onboard/probe-diagnostics.ts, src/lib/onboard/probe-diagnostics.test.ts	New module with a non-exported summarizeProbeFailureForDisplay helper and an exported summarizeProbeForDisplay that prioritizes structured failures entries and falls back to parsing message; Vitest suite asserts redaction of secrets and correct fallback classifications.
Integration into validation failure logging src/lib/onboard/inference-selection-validation.ts	Imports summarizeProbeForDisplay, extends printValidationFailure to log a "Validation probe summary" line with a credentials-omitted notice when a probe is provided, and updates all four validation failure call sites to pass the probe.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

area: cli, area: onboarding, v0.0.64

Poem

🐇 A probe has failed, but fear not, dear friend,
No secrets leak out — credentials we defend!
I sniff at the status, the curl exit too,
And summarize neatly what went askew.
Hop, hop! Clean logs for me and for you! 🌿

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 12.50% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix(onboard): summarize inference validation failures' clearly and concisely describes the main change—adding diagnostic summaries to inference validation failure messages.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/onboard-validation-error-summary

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the fix/onboard-validati... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.

File	main	fix/onboard-validati... f002451	+/-
nemoclaw/src/se...cret-scanner.ts	—	100%	—
nemoclaw/src/commands/slash.ts	—	100%	—
nemoclaw/src/li...bprocess-env.ts	—	100%	—
nemoclaw/src/bl...eprint/state.ts	—	98%	—
nemoclaw/src/onboard/config.ts	—	98%	—
nemoclaw/src/bl...int/snapshot.ts	—	97%	—
nemoclaw/src/bl...print/runner.ts	—	95%	—
nemoclaw/src/co...ration-state.ts	—	94%	—
nemoclaw/src/bl...ate-networks.ts	—	94%	—
nemoclaw/src/index.ts	—	94%	—

TypeScript / code-coverage/cli

The overall coverage in the fix/onboard-validati... branch is 44%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.

File	main	fix/onboard-validati... f002451	+/-
src/lib/state/o...oard-session.ts	—	90%	—
src/lib/inference/local.ts	—	77%	—
src/lib/sandbox/config.ts	—	72%	—
src/lib/inference/nim.ts	—	72%	—
src/lib/onboard/preflight.ts	—	64%	—
src/lib/state/sandbox.ts	—	55%	—
src/lib/onboard...er-gpu-patch.ts	—	50%	—
src/lib/actions...licy-channel.ts	—	49%	—
src/lib/policy/index.ts	—	48%	—
src/lib/onboard.ts	—	17%	—

---

_{Updated June 14, 2026 16:50 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/onboard/inference-selection-validation.ts`:
- Line 90: The console.error message on line 90 has been changed from the
original wording "Validation details were omitted to avoid exposing
credentials." to "Detailed response bodies were omitted to avoid exposing
credentials." However, downstream validation-failure classification logic and
tests depend on the exact original wording. Revert the console.error message
text back to the original canonical wording to maintain compatibility with
downstream classifiers and prevent breaking validation-failure classification.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f56fd6e3-7cc8-40d8-b837-37e2a1a27f84

📥 Commits

Reviewing files that changed from the base of the PR and between 618e978 and 17765d1.

📒 Files selected for processing (3)

• src/lib/onboard/inference-selection-validation.ts
• src/lib/onboard/probe-diagnostics.test.ts
• src/lib/onboard/probe-diagnostics.ts

[github-actions]

E2E Advisor Recommendation

Required E2E: onboard-negative-paths-vitest, inference-routing-vitest
Optional E2E: credential-sanitization-vitest, cloud-inference-vitest

Dispatch hint: onboard-negative-paths-vitest,inference-routing-vitest

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

• onboard-negative-paths-vitest (medium): Required because onboarding negative-path behavior and CLI failure output changed. This job exercises real non-interactive onboard invalid-key handling and verifies failures remain clean and stack-trace-free.
• inference-routing-vitest (high): Required because provider probe failure reporting is part of inference selection/routing. This job covers invalid API key and unreachable endpoint onboard failures, credential/transport classification, cleanup, and non-exposure of raw stack traces or submitted keys.

Optional E2E

• credential-sanitization-vitest (high): Optional adjacent confidence for the broader credential non-exposure boundary, but the changed code is CLI probe diagnostic output rather than sandbox filesystem/config credential sanitization.
• cloud-inference-vitest (high): Optional positive-path confidence that normal cloud onboarding and inference still work after adding probe diagnostic handling, though the PR primarily affects failed validation output.

New E2E recommendations

• onboarding probe diagnostics redaction (medium): Existing E2E negative-path coverage checks classification and secret non-exposure, but does not appear to assert the new 'Validation probe summary' contract against a provider response body containing sensitive text.
  • Suggested test: Add a focused onboard/inference-routing E2E case using a controlled OpenAI-compatible mock endpoint that returns an HTTP error body with secret-shaped content, then assert CLI output includes only the coarse HTTP/curl probe summary and excludes the raw body.

Dispatch hint

• Workflow: .github/workflows/e2e-vitest-scenarios.yaml
• jobs input: onboard-negative-paths-vitest,inference-routing-vitest

[github-actions]

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: ubuntu-repo-cloud-openclaw
Optional Vitest E2E scenarios: None

Dispatch required Vitest E2E scenarios:

• gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: medium

Required Vitest E2E scenarios

• ubuntu-repo-cloud-openclaw: The PR changes onboarding endpoint validation diagnostics used when probing inference providers. The smallest live-supported Vitest scenario that exercises the affected cloud OpenClaw onboarding path is ubuntu-repo-cloud-openclaw.
  • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Optional Vitest E2E scenarios

• None.

Relevant changed files

• src/lib/onboard/inference-selection-validation.ts
• src/lib/onboard/probe-diagnostics.ts

[github-actions]

PR Review Advisor

Findings: 0 needs attention, 2 worth checking, 0 nice ideas
Since last review: 0 prior items resolved, 4 still apply, 0 new items found

Review findings

🛠️ Needs attention

• None.

🔎 Worth checking

• Claimed stderr redaction coverage is not explicit (src/lib/onboard/probe-diagnostics.test.ts:27): The PR claims tests prove raw provider bodies, stderr, and credential-shaped values are not surfaced in summaries. The added tests cover raw provider body and credential-shaped values in body/message fields, and they cover curl message redaction, but no test fixture includes a stderr field. The implementation currently ignores stderr, so this is a test-evidence gap rather than a confirmed leak.
  • Recommendation: Add a regression case where a failure includes stderr containing raw provider output and a credential-shaped value, then assert summarizeProbeForDisplay() does not include that stderr content.
  • Evidence: The new curl test at src/lib/onboard/probe-diagnostics.test.ts uses message with secret text but no stderr property; summarizeProbeForDisplay() intentionally only reads failures[].name, httpStatus, curlStatus, and probe.message.
• Caller logging paths still lack mocked redaction coverage (src/lib/onboard/inference-selection-validation.ts:88): The summarizer unit tests are good, but the changed callers sit on credential retrieval, provider probe, console logging, process-exit, and recovery-prompt boundaries. A mocked integration test would verify the helper is wired into each validation failure path and that raw probe details are not accidentally printed by the surrounding control flow.
  • Recommendation: Add focused mocked tests for OpenAI-like, Anthropic, and noninteractive validation failures that inject secret-bearing message/body/stderr values and assert console.error receives only the endpoint failure line, coarse Validation probe summary, and redaction notice.
  • Evidence: All four failure branches now call printValidationFailure(label, probe), but only src/lib/onboard/probe-diagnostics.test.ts directly tests summarizeProbeForDisplay(); no test exercises createInferenceSelectionValidationHelpers() logging behavior.

🌱 Nice ideas

• None.

Consider writing more tests for

• **Mocked behavioral coverage** — validateOpenAiLikeSelection with a mocked failed probe containing secret-bearing message, body, and stderr logs only the endpoint failure line, coarse Validation probe summary, and redaction notice, then passes the full probe to promptValidationRecovery.. The new helper has useful unit tests, but the changed validation code performs console logging across credential, network-probe, recovery-prompt, and process-exit boundaries.
• **Mocked behavioral coverage** — validateAnthropicSelectionWithRetryMessage with a mocked failed probe containing raw provider text and credential-shaped values logs the sanitized summary before prompting recovery and never logs raw response text.. The new helper has useful unit tests, but the changed validation code performs console logging across credential, network-probe, recovery-prompt, and process-exit boundaries.
• **Mocked behavioral coverage** — Noninteractive OpenAI-like validation failure logs the sanitized summary and redaction notice, calls process.exit(1), and does not invoke promptValidationRecovery.. The new helper has useful unit tests, but the changed validation code performs console logging across credential, network-probe, recovery-prompt, and process-exit boundaries.
• **Mocked behavioral coverage** — summarizeProbeForDisplay with failure.stderr set to raw provider output containing a credential-shaped value does not include stderr content in the returned summary.. The new helper has useful unit tests, but the changed validation code performs console logging across credential, network-probe, recovery-prompt, and process-exit boundaries.
• **Claimed stderr redaction coverage is not explicit** — Add a regression case where a failure includes stderr containing raw provider output and a credential-shaped value, then assert summarizeProbeForDisplay() does not include that stderr content.
• **Caller logging paths still lack mocked redaction coverage** — Add focused mocked tests for OpenAI-like, Anthropic, and noninteractive validation failures that inject secret-bearing message/body/stderr values and assert console.error receives only the endpoint failure line, coarse Validation probe summary, and redaction notice.
• **Acceptance clause:** Add tests proving raw provider bodies, stderr, and credential-shaped values are not surfaced in summaries. — add test evidence or identify existing coverage. src/lib/onboard/probe-diagnostics.test.ts covers raw provider body and credential-shaped strings in message/body inputs, plus curl message redaction. No test fixture includes a stderr field, so that named part of the clause is not explicitly proven.

Since last review details

Current findings:

• Claimed stderr redaction coverage is not explicit (src/lib/onboard/probe-diagnostics.test.ts:27): The PR claims tests prove raw provider bodies, stderr, and credential-shaped values are not surfaced in summaries. The added tests cover raw provider body and credential-shaped values in body/message fields, and they cover curl message redaction, but no test fixture includes a stderr field. The implementation currently ignores stderr, so this is a test-evidence gap rather than a confirmed leak.
  • Recommendation: Add a regression case where a failure includes stderr containing raw provider output and a credential-shaped value, then assert summarizeProbeForDisplay() does not include that stderr content.
  • Evidence: The new curl test at src/lib/onboard/probe-diagnostics.test.ts uses message with secret text but no stderr property; summarizeProbeForDisplay() intentionally only reads failures[].name, httpStatus, curlStatus, and probe.message.
• Caller logging paths still lack mocked redaction coverage (src/lib/onboard/inference-selection-validation.ts:88): The summarizer unit tests are good, but the changed callers sit on credential retrieval, provider probe, console logging, process-exit, and recovery-prompt boundaries. A mocked integration test would verify the helper is wired into each validation failure path and that raw probe details are not accidentally printed by the surrounding control flow.
  • Recommendation: Add focused mocked tests for OpenAI-like, Anthropic, and noninteractive validation failures that inject secret-bearing message/body/stderr values and assert console.error receives only the endpoint failure line, coarse Validation probe summary, and redaction notice.
  • Evidence: All four failure branches now call printValidationFailure(label, probe), but only src/lib/onboard/probe-diagnostics.test.ts directly tests summarizeProbeForDisplay(); no test exercises createInferenceSelectionValidationHelpers() logging behavior.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.