Skip to content

fix(onboard): smoke test inference before success #3603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jyaunches merged 10 commits into from
May 15, 2026
Merged

fix(onboard): smoke test inference before success #3603

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
c108c07
fix(onboard): smoke test inference before success
jyaunches May 15, 2026
57769e4
fix(onboard): extract inference smoke helper
jyaunches May 15, 2026
b91f668
fix(onboard): keep inference smoke out of entrypoint
jyaunches May 15, 2026
6ee069f
fix(onboard): move smoke probe into inference probes
jyaunches May 15, 2026
4c3bd0f
fix(onboard): keep entrypoint budget neutral
jyaunches May 15, 2026
68d9be9
fix(onboard): avoid logging credential env name
jyaunches May 15, 2026
33b3136
Merge remote-tracking branch 'origin/main' into fix-3253-onboard-infe…
jyaunches May 15, 2026
9af9c3b
Merge remote-tracking branch 'origin/main' into fix-3253-onboard-infe…
jyaunches May 15, 2026
ef02afb
Merge remote-tracking branch 'origin/main' into fix-3253-onboard-infe…
jyaunches May 15, 2026
70144c6
Merge remote-tracking branch 'origin/main' into fix-3253-onboard-infe…
jyaunches May 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion src/lib/inference/onboard-probes.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
// Inference endpoint probes — validate that a provider's API responds
// before committing the onboard wizard to a model selection.

const { normalizeCredentialValue } = require("../credentials/store");
const { getCredential, normalizeCredentialValue, resolveProviderCredential } = require("../credentials/store");
const { isWsl } = require("../platform");
const httpProbe = require("../adapters/http/probe");
const {
Expand Down Expand Up @@ -804,3 +804,44 @@ module.exports = {
probeAnthropicEndpoint,
RETRIABLE_HTTP_PROBE_STATUSES,
};

function shouldSmokeOpenAiLikeOnboardRoute(provider) {
const { REMOTE_PROVIDER_CONFIG } = require("../onboard/providers");
if (provider === "nvidia-nim" || provider === "nvidia-router") return true;
return Object.values(REMOTE_PROVIDER_CONFIG).some(
(entry) => entry.providerName === provider && entry.providerType === "openai",
);
}

function verifyOnboardInferenceSmoke(options) {
if (!shouldSmokeOpenAiLikeOnboardRoute(options.provider)) return;
if (process.env.VITEST === "true") return;

const endpointUrl = options.endpointUrl || require("./config").INFERENCE_ROUTE_URL;
const credentialEnv = options.credentialEnv || null;
const apiKey = credentialEnv
? resolveProviderCredential(credentialEnv) || getCredential(credentialEnv) || ""
: "";
const probe = probeOpenAiLikeEndpoint(endpointUrl, options.model, apiKey, {
authMode: getProbeAuthMode(options.provider),
skipResponsesProbe: true,
});

if (probe.ok) {
console.log(` ✓ Inference smoke passed: ${options.provider} / ${options.model}`);
return;
}

const { compactText } = require("../core/url-utils");
const { redact } = require("../runner");
console.error(" Onboard inference smoke check failed.");
console.error(` Provider: ${options.provider}`);
console.error(` Model: ${options.model}`);
console.error(` API base: ${endpointUrl}`);
if (credentialEnv) console.error(" Credential env: configured");
console.error(` Upstream error: ${compactText(redact(probe.message || "unknown inference failure"))}`);
Comment on lines +835 to +842

@coderabbitai coderabbitai Bot May 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Redact API base in smoke-failure diagnostics.

Line 840 logs endpointUrl raw. If the configured base URL contains embedded credentials (query param or userinfo), this leaks secrets to console/log capture. Redact before printing.

Suggested patch 
   const { compactText } = require("../core/url-utils");
   const { redact } = require("../runner");
+  const safeEndpointUrl = compactText(redact(String(endpointUrl || "")));
   console.error("  Onboard inference smoke check failed.");
   console.error(`  Provider: ${options.provider}`);
   console.error(`  Model: ${options.model}`);
-  console.error(`  API base: ${endpointUrl}`);
+  console.error(`  API base: ${safeEndpointUrl}`);
   if (credentialEnv) console.error(`  Credential env: ${credentialEnv}`);
   console.error(`  Upstream error: ${compactText(redact(probe.message || "unknown inference failure"))}`);
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const { compactText } = require("../core/url-utils");
const { redact } = require("../runner");
console.error(" Onboard inference smoke check failed.");
console.error(` Provider: ${options.provider}`);
console.error(` Model: ${options.model}`);
console.error(` API base: ${endpointUrl}`);
if (credentialEnv) console.error(` Credential env: ${credentialEnv}`);
console.error(` Upstream error: ${compactText(redact(probe.message || "unknown inference failure"))}`);
const { compactText } = require("../core/url-utils");
const { redact } = require("../runner");
const safeEndpointUrl = compactText(redact(String(endpointUrl || "")));
console.error(" Onboard inference smoke check failed.");
console.error(` Provider: ${options.provider}`);
console.error(` Model: ${options.model}`);
console.error(` API base: ${safeEndpointUrl}`);
if (credentialEnv) console.error(` Credential env: ${credentialEnv}`);
console.error(` Upstream error: ${compactText(redact(probe.message || "unknown inference failure"))}`);
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/inference/onboard-probes.ts` around lines 835 - 842, The smoke-check
logging prints endpointUrl raw and may leak credentials; update the log to
redact and compact the URL before printing by passing endpointUrl through
redact() and compactText() like the upstream error, i.e., replace the raw
endpointUrl usage in the Onboard probes logging block that references
endpointUrl with compactText(redact(endpointUrl || "unknown API base")) so the
diagnostics show a redacted URL while preserving existing
provider/model/credentialEnv logging and behavior.

process.exit(1);
}

module.exports.shouldSmokeOpenAiLikeOnboardRoute = shouldSmokeOpenAiLikeOnboardRoute;
module.exports.verifyOnboardInferenceSmoke = verifyOnboardInferenceSmoke;
7 changes: 3 additions & 4 deletions src/lib/onboard.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2367,16 +2367,13 @@ const {
hasChatCompletionsToolCall,
hasChatCompletionsToolCallLeak,
shouldRequireResponsesToolCalling,
verifyOnboardInferenceSmoke,
getProbeAuthMode,
getValidationProbeCurlArgs,
probeOpenAiLikeEndpoint,
probeAnthropicEndpoint,
} = require("./inference/onboard-probes");

// shouldSkipResponsesProbe and isNvcfFunctionNotFoundForAccount /
// nvcfFunctionNotFoundMessage — see validation import above. They live in
// src/lib/validation.ts so they can be unit-tested independently.

async function validateOpenAiLikeSelection(
label: string,
endpointUrl: string,
Expand Down Expand Up @@ -7708,6 +7705,7 @@ async function setupInference(
}

verifyInferenceRoute(provider, model);
verifyOnboardInferenceSmoke({ provider, model, endpointUrl, credentialEnv });

@coderabbitai coderabbitai Bot May 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Run the smoke probe on the resume fast-path too.

These probes run in setupInference, but onboard() can skip that function when resume && isInferenceRouteReady(provider, model), which allows a resumed run to report success without executing the new smoke validation.

Suggested fix 
       if (isRoutedInferenceProvider(provider)) {
         try {
           await reconcileModelRouter();
         } catch (err) {
           console.error(
             `  ✗ Failed to reconcile model router: ${err instanceof Error ? err.message : String(err)}`,
           );
           process.exit(1);
         }
       }
       skippedStepMessage("inference", `${provider} / ${model}`);
+      verifyOnboardInferenceSmoke({ provider, model, endpointUrl, credentialEnv });
       if (nimContainer && sandboxName) {
         registry.updateSandbox(sandboxName, { nimContainer });
       }

Also applies to: 7989-7989

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard.ts` at line 7713, When the onboarding code takes the resume
fast-path (when resume && isInferenceRouteReady(provider, model)) it skips
setupInference and therefore skips the smoke probe; ensure
verifyOnboardInferenceSmoke({ provider, model, endpointUrl, credentialEnv }) is
invoked in the onboard resume fast-path before returning success (i.e., add the
same smoke validation call to the branch guarded by resume &&
isInferenceRouteReady in the onboard function), and mirror the same insertion
for the other resume-fast-path occurrence referenced (the other
isInferenceRouteReady/resume branch) so resumed runs run the smoke probe as
well.

if (sandboxName) {
registry.updateSandbox(sandboxName, { model, provider });
}
Expand Down Expand Up @@ -7983,6 +7981,7 @@ async function setupInference(
}

verifyInferenceRoute(provider, model);
verifyOnboardInferenceSmoke({ provider, model, endpointUrl, credentialEnv });
if (sandboxName) {
registry.updateSandbox(sandboxName, { model, provider });
}
Expand Down
Loading