fix: support reasoning compatible endpoints (Fixes #3279)

[deepujain]

Summary

Custom OpenAI-compatible providers can opt into reasoning-mode validation through NEMOCLAW_REASONING. When that flag is enabled, NemoClaw skips the Responses/tool-call probe that reasoning-only wrappers often reject, and the sandbox smoke check accepts reasoning text as valid output.

Fixes #3279.

Changes

• Normalized NEMOCLAW_REASONING aliases such as yes, 1, no, and 0.
• Persisted compatible-endpoint reasoning state through onboarding session updates and provider resume.
• Skipped Responses/tool-call and streaming probes for reasoning-mode compatible endpoints.
• Increased the sandbox smoke response budget and accepted reasoning_content or reasoning when message.content is empty.
• Added focused tests for flag normalization, provider state handoff, session persistence, and the custom endpoint reasoning path.

Testing

• npm run build:cli passed.
• npm run typecheck:cli passed.
• npm test -- test/onboard-selection.test.ts -t "honors NEMOCLAW_REASONING" passed.
• npm test -- src/lib/onboard/machine/handlers/provider-inference.test.ts src/lib/state/onboard-session.test.ts src/lib/onboard/compatible-endpoint-smoke.test.ts passed.
• HOME=/private/tmp/nemoclaw-test-home-3279-rebase npm test -- --reporter=dot was attempted; it timed out locally after 180s before completing.

Evidence it works

The focused custom endpoint test simulates a reasoning-only chat response with empty content and non-empty reasoning_content. With NEMOCLAW_REASONING=yes, onboarding accepts the provider, avoids /responses and streaming probes, and keeps preferredInferenceApi on chat completions. Session tests cover persisting and clearing the reasoning flag during provider handoff and resume.

Signed-off-by: Deepak Jain deepujain@gmail.com

Summary by CodeRabbit

• New Features
  • Added support for a compatible-endpoint reasoning mode during onboarding, with the chosen setting preserved across sessions and restored when resuming setup.
• Bug Fixes
  • Improved validation for custom OpenAI-compatible endpoints so onboarding adapts correctly when reasoning features are enabled.
  • Enhanced endpoint smoke checks to better handle different response formats and larger outputs.
• Documentation
  • Updated platform and provider reference notes to match the latest onboarding behavior.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a NEMOCLAW_REASONING reasoning-mode flag for the "Other OpenAI-compatible endpoint" provider. Two new helpers normalize and apply the flag via process.env. The onboarding session gains a compatibleEndpointReasoning field. Validation probes and the sandbox smoke test branch on the flag. Doc and CI line references are updated to reflect shifted line numbers.

Changes

Reasoning mode for OpenAI-compatible endpoints

Layer / File(s)	Summary
Reasoning helpers and exports src/lib/onboard.ts	normalizeReasoningFlag() maps truthy/falsy aliases to "true", "false", or null; configureCompatibleEndpointReasoning() sets or clears process.env.NEMOCLAW_REASONING and returns the resolved value. Both are exported.
Session persistence for reasoning flag src/lib/state/onboard-session.ts	Session and SessionUpdates gain compatibleEndpointReasoning; createSession(), normalizeSession(), filterSafeUpdates(), and summarizeForDebug() are updated to read, write, whitelist, and surface the field.
Provider selection, resume, and probe wiring src/lib/onboard.ts	configureCompatibleEndpointReasoning() is called during custom-provider selection and onboarding resume; the result flows through toSessionUpdates() and setupNim() return; validateCustomOpenAiLikeSelection() branches tool-calling requirements, responses-probe inclusion, and streaming on NEMOCLAW_REASONING === "true".
Sandbox smoke probe src/lib/onboard.ts	max_tokens raised to 512; response text is extracted from message.content with fallbacks to message.reasoning_content and message.reasoning for reasoning-only models.
Doc and CI line reference updates ci/platform-matrix.json, docs/inference/inference-options.mdx, docs/reference/platform-support.mdx	Embedded src/lib/onboard.ts line-number citations updated to reflect shifted line numbers after new code additions.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• NVIDIA/NemoClaw#1956: Propagates NEMOCLAW_REASONING and token/context env vars into the sandbox image build via patchStagedDockerfile(), directly connected to the env-var wiring added here.
• NVIDIA/NemoClaw#5932: Modifies the compatible-endpoint smoke flow to retry transient failures, touching the same probe execution path adjusted in this PR.

Suggested labels

area: onboarding, area: sandbox

Suggested reviewers

• cv
• jyaunches
• ericksoa

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: reasoning support for compatible endpoints.
Linked Issues check	✅ Passed	The changes implement the #3279 requirements by adding runtime reasoning support, skipping incompatible probes, and handling reasoning-style responses.
Out of Scope Changes check	✅ Passed	The documentation and matrix edits stay aligned with the onboarding reasoning fix and do not introduce unrelated scope.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

test/onboard-selection.test.ts (1)

2075-2094: ⚡ Quick win

Assert probe path behavior explicitly in reasoning mode.

Line [2077] and Line [2150] currently validate end-state only; this test can still pass if /responses (or streaming probe) is called and then fallback succeeds. Please log curl args and assert reasoning mode avoids /responses/-N while still hitting /chat/completions.

Suggested test hardening

@@
     const scriptPath = path.join(tmpDir, "custom-openai-reasoning-check.js");
+    const curlArgsLog = path.join(tmpDir, "custom-openai-reasoning-curl-args.log");
@@
       path.join(fakeBin, "curl"),
       `#!/usr/bin/env bash
+args_log=${JSON.stringify(curlArgsLog)}
+printf '%s\\n' "$*" >> "$args_log"
 body='{"error":{"message":"bad request"}}'
 status="400"
 outfile=""
 url=""
@@
     assert.equal(payload.result.model, "reasoning-model");
     assert.equal(payload.result.preferredInferenceApi, "openai-completions");
     assert.equal(payload.reasoning, "true");
+    const curlInvocations = fs.readFileSync(curlArgsLog, "utf-8");
+    assert.match(curlInvocations, /chat\/completions/);
+    assert.doesNotMatch(curlInvocations, /\/responses/);
+    assert.doesNotMatch(curlInvocations, /(^|\s)-N(\s|$)/);
     assert.ok(
       payload.messages.every(
         (message: string) => !/Enable reasoning mode for this model/.test(message),
       ),
     );

Also applies to: 2150-2160

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/onboard-selection.test.ts` around lines 2075 - 2094, The fake curl
script written to path.join(fakeBin, "curl") should record its received
arguments so the test can assert probe behavior: modify the heredoc to append
the full "$@" or each processed arg into a temp log file (e.g.,
"$outfile.args.log" or a known capture file in the test dir) before handling
-o/url logic, and keep returning the same body/status logic; then update the
test assertions to read that captured args log and explicitly assert that in
reasoning mode the captured args do not include '/responses' nor the '-N' flag
and do include a call to '/chat/completions'. Ensure you reference the same
fakeBin/curl generation code and the test's reasoning-mode invocation when
adding the new assertions.

test/onboard.test.ts (1)

391-410: ⚡ Quick win

Add an explicit assertion for the unset/default reasoning path.

This test validates aliases, but not the key default behavior when NEMOCLAW_REASONING is unset. Locking that in will better guard the PR’s intended fallback semantics.

✅ Suggested test addition

   expect(normalizeReasoningFlag("maybe")).toBeNull();

+  delete process.env.NEMOCLAW_REASONING;
+  await expect(configureCompatibleEndpointReasoning()).resolves.toBe("false");
+  expect(process.env.NEMOCLAW_REASONING).toBe("false");
+
   process.env.NEMOCLAW_REASONING = "yes";
   await expect(configureCompatibleEndpointReasoning()).resolves.toBe("true");
   expect(process.env.NEMOCLAW_REASONING).toBe("true");

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/onboard.test.ts` around lines 391 - 410, Add an explicit assertion that
the "unset/default" path is covered: before setting NEMOCLAW_REASONING to "yes",
delete process.env.NEMOCLAW_REASONING (or set it to undefined) and call await
expect(configureCompatibleEndpointReasoning()).resolves.toBe(normalizeReasoningFlag(undefined));
this ensures the test asserts the function's fallback behavior when
NEMOCLAW_REASONING is not present and references the existing
normalizeReasoningFlag and configureCompatibleEndpointReasoning symbols.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/onboard.ts`:
- Around line 1508-1511: The configureCompatibleEndpointReasoning function
currently stores the normalized NEMOCLAW_REASONING in process.env only, causing
stale or missing values across onboarding flows; update it to persist the
normalized value into the selected provider state object (use the same
normalization via normalizeReasoningFlag) instead of relying solely on
process.env, and ensure that when provider selection changes the stored flag is
cleared from that provider state and process.env is restored/updated
accordingly; locate configureCompatibleEndpointReasoning and related calls to
normalizeReasoningFlag and modify the provider selection/restore logic where
provider state is mutated so the flag is saved with the provider, removed on
selection change, and used to seed process.env when a provider is active (also
apply the same change pattern to the other occurrence referenced near the file's
later provider-state handling).

---

Nitpick comments:
In `@test/onboard-selection.test.ts`:
- Around line 2075-2094: The fake curl script written to path.join(fakeBin,
"curl") should record its received arguments so the test can assert probe
behavior: modify the heredoc to append the full "$@" or each processed arg into
a temp log file (e.g., "$outfile.args.log" or a known capture file in the test
dir) before handling -o/url logic, and keep returning the same body/status
logic; then update the test assertions to read that captured args log and
explicitly assert that in reasoning mode the captured args do not include
'/responses' nor the '-N' flag and do include a call to '/chat/completions'.
Ensure you reference the same fakeBin/curl generation code and the test's
reasoning-mode invocation when adding the new assertions.

In `@test/onboard.test.ts`:
- Around line 391-410: Add an explicit assertion that the "unset/default" path
is covered: before setting NEMOCLAW_REASONING to "yes", delete
process.env.NEMOCLAW_REASONING (or set it to undefined) and call await
expect(configureCompatibleEndpointReasoning()).resolves.toBe(normalizeReasoningFlag(undefined));
this ensures the test asserts the function's fallback behavior when
NEMOCLAW_REASONING is not present and references the existing
normalizeReasoningFlag and configureCompatibleEndpointReasoning symbols.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 78ba4585-9297-4c89-b6ae-5ad64ecf87cb

📥 Commits

Reviewing files that changed from the base of the PR and between f1568f6 and 077cb4b.

📒 Files selected for processing (3)

• src/lib/onboard.ts
• test/onboard-selection.test.ts
• test/onboard.test.ts

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/lib/state/onboard-session.ts (1)

707-717: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Persist compatibleEndpointReasoning in the safe update whitelist.

SessionUpdates adds this field (Line 131), but filterSafeUpdates() never forwards it, so updates passed via markStepComplete() / completeSession() are silently dropped. That breaks both setting and clearing (null) this value during provider transitions.

Suggested fix

 export function filterSafeUpdates(updates: SessionUpdates): Partial<Session> {
   const safe: Partial<Session> = {};
   if (!isObject(updates)) return safe;
   if (typeof updates.sandboxName === "string") safe.sandboxName = updates.sandboxName;
   if (typeof updates.provider === "string") safe.provider = updates.provider;
   if (typeof updates.model === "string") safe.model = updates.model;
   if (typeof updates.endpointUrl === "string") safe.endpointUrl = redactUrl(updates.endpointUrl);
   if (typeof updates.credentialEnv === "string") safe.credentialEnv = updates.credentialEnv;
   if (typeof updates.preferredInferenceApi === "string")
     safe.preferredInferenceApi = updates.preferredInferenceApi;
+  if (typeof updates.compatibleEndpointReasoning === "string") {
+    safe.compatibleEndpointReasoning = updates.compatibleEndpointReasoning;
+  } else if (updates.compatibleEndpointReasoning === null) {
+    safe.compatibleEndpointReasoning = null;
+  }
   if (typeof updates.nimContainer === "string") safe.nimContainer = updates.nimContainer;

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/state/onboard-session.ts` around lines 707 - 717, filterSafeUpdates
currently omits the SessionUpdates field compatibleEndpointReasoning so updates
from markStepComplete/completeSession are dropped; update the function
filterSafeUpdates to accept and propagate compatibleEndpointReasoning (including
allowing null to clear it) into the returned Partial<Session> when typeof
updates.compatibleEndpointReasoning is "string" or when
updates.compatibleEndpointReasoning === null, mirroring how other
nullable/optional fields are handled so provider transitions can set and clear
this value.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/lib/state/onboard-session.ts`:
- Around line 707-717: filterSafeUpdates currently omits the SessionUpdates
field compatibleEndpointReasoning so updates from
markStepComplete/completeSession are dropped; update the function
filterSafeUpdates to accept and propagate compatibleEndpointReasoning (including
allowing null to clear it) into the returned Partial<Session> when typeof
updates.compatibleEndpointReasoning is "string" or when
updates.compatibleEndpointReasoning === null, mirroring how other
nullable/optional fields are handled so provider transitions can set and clear
this value.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 06d7e88a-98bf-4a0d-89a9-ddd0edb6c09a

📥 Commits

Reviewing files that changed from the base of the PR and between 077cb4b and 66bbe29.

📒 Files selected for processing (4)

• src/lib/onboard.ts
• src/lib/state/onboard-session.ts
• test/onboard-selection.test.ts
• test/onboard.test.ts

🚧 Files skipped from review as they are similar to previous changes (3)

• test/onboard-selection.test.ts
• test/onboard.test.ts
• src/lib/onboard.ts

[coderabbitai]

🧹 Nitpick comments (1)

src/lib/state/onboard-session.ts (1)

145-166: ⚡ Quick win

Include compatibleEndpointReasoning in debug summary output.

The new persisted field is not exposed by DebugSessionSummary/summarizeForDebug, which makes reasoning-mode resume issues harder to diagnose from debug dumps.

Proposed patch

 export interface DebugSessionSummary {
   version: number;
   sessionId: string;
   status: string;
   resumable: boolean;
   mode: string;
   startedAt: string;
   updatedAt: string;
   sandboxName: string | null;
   provider: string | null;
   model: string | null;
   endpointUrl: string | null;
   credentialEnv: string | null;
   preferredInferenceApi: string | null;
+  compatibleEndpointReasoning: string | null;
   nimContainer: string | null;
   policyPresets: string[] | null;
   gpuPassthrough: boolean;
   lastStepStarted: string | null;
   lastCompletedStep: string | null;
   failure: SessionFailure | null;
   steps: Record<string, StepState>;
 }
@@
     endpointUrl: redactUrl(session.endpointUrl),
     credentialEnv: session.credentialEnv,
     preferredInferenceApi: session.preferredInferenceApi,
+    compatibleEndpointReasoning: session.compatibleEndpointReasoning,
     nimContainer: session.nimContainer,
     policyPresets: session.policyPresets,

Also applies to: 852-867

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/state/onboard-session.ts` around lines 145 - 166, DebugSessionSummary
and the summarizeForDebug output are missing the new persisted field
compatibleEndpointReasoning, so add compatibleEndpointReasoning: string | null
(matching the persisted type) to the DebugSessionSummary interface and update
the summarizeForDebug function to read the session-compatibleEndpointReasoning
value and include it in the returned summary object (alongside fields like
failure, steps, nimContainer, preferredInferenceApi, etc.) so debug dumps expose
reasoning-mode compatibility; update any related spots mentioned (e.g., the
summarizeForDebug implementation around the lastStep/lastCompletedStep logic) to
populate this field.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/state/onboard-session.ts`:
- Around line 145-166: DebugSessionSummary and the summarizeForDebug output are
missing the new persisted field compatibleEndpointReasoning, so add
compatibleEndpointReasoning: string | null (matching the persisted type) to the
DebugSessionSummary interface and update the summarizeForDebug function to read
the session-compatibleEndpointReasoning value and include it in the returned
summary object (alongside fields like failure, steps, nimContainer,
preferredInferenceApi, etc.) so debug dumps expose reasoning-mode compatibility;
update any related spots mentioned (e.g., the summarizeForDebug implementation
around the lastStep/lastCompletedStep logic) to populate this field.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f8e80300-8c3d-41f3-be29-8da2e7314cd8

📥 Commits

Reviewing files that changed from the base of the PR and between 66bbe29 and 2c1672b.

📒 Files selected for processing (1)

• src/lib/state/onboard-session.ts

[wscurran]

✨ Thanks for submitting this detailed PR about supporting reasoning-compatible endpoints for custom OpenAI providers. This change aims to improve the compatibility of NemoClaw with reasoning-mode validation through the NEMOCLAW_REASONING flag.

---

Related open issues:

• #3279 NEMOCLAW_REASONING not configurable for Option 3 providers; reasoning-only models fail silently

[deepujain]

Rebased on current main and trimmed the accidental test-file expansion so the diff stays focused. build:cli plus focused reasoning/session/provider tests pass.

[deepujain]

Follow-up: moved the reasoning-mode helpers under src/lib/onboard/ so the top-level onboard entrypoint is net-neutral for the growth guard. build:cli and the focused reasoning/session/provider tests pass.

[deepujain]

Rebased onto current main and kept the onboard entrypoint net-neutral. build:cli plus the focused reasoning, session, provider, and onboard-selection tests pass.

[deepujain]

Rebased on current main, re-signed the commit stack, and cleaned up the PR body DCO sign-off. Build, typecheck, and focused reasoning/session tests pass locally. Ready for another look.

[deepujain]

Follow-up: fixed the growth guard by keeping the top-level onboard entrypoint net-negative against main. Build, typecheck, and the focused reasoning/session tests still pass locally.

[deepujain]

One more cleanup: moved the reasoning selection coverage into a focused test file so the legacy test-size budget stays at its limit. Build, typecheck, and the focused tests pass locally.

[deepujain]

Rebased on current main and carried compatible reasoning through the core onboarding flow context. Build, typecheck, lint, source-shape, and focused reasoning/session/onboarding tests pass locally.

[deepujain]

Rebased onto current main and carried the compatible-endpoint reasoning state through the migrated provider/flow helpers. Build, typecheck, lint, source-shape, and focused reasoning/provider-flow tests pass.

[deepujain]

Fixed the growth-guardrail failure as well: the onboard entrypoint is now net-neutral for this PR while keeping the reasoning helper behavior in the extracted module. Re-ran build, typecheck, lint, source-shape, and the focused reasoning/provider-flow tests.

[deepujain]

Rebased this on current main and kept the reasoning coverage on the current package-contract boundary. Build, typecheck, lint, source-shape, and the focused reasoning/provider-flow tests pass.

[deepujain]

Closing this as superseded now that #3279 is closed and current main already carries the behavior. The remaining branch diff is only leftover test coverage, so keeping it open would add churn without a clear issue to close.