fix(onboard): accept reasoning-mode models in the inference smoke probe

[latenighthackathon]

Summary

Closes #3341. The Option-3 compatible-endpoint smoke probe sent max_tokens=32 and required choices[0].message.content to be a non-empty string. Thinking-mode models like Qwen3.6 in vLLM with --reasoning-parser qwen3 exhaust the entire 32-token budget on the reasoning chain and return content=null with finish_reason="length", even though the endpoint is healthy and the round-trip succeeded. Onboard then rejected the response and exited at step [7/8] Validating inference.

Related Issue

Closes #3341

Changes

Two changes in the embedded smoke script in src/lib/onboard.ts:

• Bump max_tokens from 32 to 256 so short reasoning chains have room to finish before content starts. 256 is still a cheap probe (~$0.001 on most paid providers for a single "say PONG") but gives a thinking model enough headroom to emit a content token.
• Treat a non-empty message.reasoning or message.reasoning_content as a valid smoke signal when content is null or empty. vLLM emits reasoning (per the qwen3 reasoning parser, observed in the Cannot use thinking models like Qwen 3.6 27B - Nemoclaw validation request token length too small. #3341 trace) and OpenAI o1-style endpoints emit reasoning_content; both indicate the endpoint round-tripped a chat completion. The probe is a liveness check, not a response-shape contract, so a reasoning-only response is enough to declare the model reachable. The success line becomes INFERENCE_SMOKE_OK (reasoning-only response, N chars) so the distinction is visible in onboard logs.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Verification

• npx prek run --all-files passes for the staged files.
• Two new test cases in test/onboard.test.ts (#3341): one asserts the bumped max_tokens is in the generated script, one asserts the reasoning-fallback branch is present.
• npx vitest run -t 'smoke|reasoning|max_tokens' test/onboard.test.ts — 6/6 pass (4 existing smoke tests + the 2 new ones).
• Embedded Python parser exercised against four sample payloads (normal content, reasoning-only Qwen3.6 shape, reasoning_content o1 shape, empty response). The first three pass with INFERENCE_SMOKE_OK, the empty payload fails as expected. Output:

[OK] normal content -> exit=0: INFERENCE_SMOKE_OK PONG
[OK] reasoning only (Qwen3.6 case) -> exit=0: INFERENCE_SMOKE_OK (reasoning-only response, 11 chars)
[OK] reasoning_content (o1 style) -> exit=0: INFERENCE_SMOKE_OK (reasoning-only response, 3 chars)
[FAIL] empty response (should fail) -> exit=1: FAIL: no content or reasoning

• npm run build:cli clean.
• No secrets, API keys, or credentials committed.
• Tests added for new behavior.

Rebased on current upstream/main (commit eb15e55e).

---

Signed-off-by: latenighthackathon latenighthackathon@users.noreply.github.com

Summary by CodeRabbit

• Tests

  • Added targeted tests for the endpoint smoke script to verify robust response handling and reasoning-mode signals.

• Improvements

  • Increased token allocation for smoke probes.
  • Strengthened response validation to tolerate malformed or reasoning-only outputs and avoid crashes on empty or unexpected structures.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The smoke probe for compatible-endpoint validation now budgets higher token limits and tolerates reasoning-only responses by accepting non-empty reasoning fields. Response parsing is defensive against malformed choices arrays. Tests cover token budget, reasoning acceptance, and safe choices handling.

Changes

Thinking Model Compatibility

Layer / File(s)	Summary
Smoke Probe Token Budget src/lib/onboard/compatible-endpoint-smoke.ts	Embedded Python payload increases max_tokens from 32 to 256 with comments explaining this prevents reasoning-chain models from exhausting budget and returning empty/length-terminated content.
Response Validator Logic src/lib/onboard/compatible-endpoint-smoke.ts	Validation logic rewrites to defensively check choices shape and presence of message dict, then accepts success when either message.content is a non-empty string or when message.reasoning/message.reasoning_content is non-empty; failure cases emit detailed error and exit non-zero.
Test Coverage src/lib/onboard/compatible-endpoint-smoke.test.ts	Three new tests verify the increased max_tokens for reasoning models, confirm reasoning-only responses are accepted as valid smoke signals with fallback field lookup and Python string validation, and ensure safe handling of empty or malformed choices arrays without unsafe indexing.

Estimated Code Review Effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 A thinking model needed time to dream,
With budget bloated to 256 tokens' beam,
When reasoning flows where content lies still,
Our validator accepts with humble will,
No crashes on choices malformed or bare—
Just graceful checks with defensive care! 🌙

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately reflects the main change: modifying the inference smoke probe to accept reasoning-mode models by increasing max_tokens and tolerating reasoning-only responses.
Linked Issues check	✅ Passed	The PR fully addresses the coding requirements in #3341: increases max_tokens from 32 to 256, validates choices shape defensively, and treats non-empty message.reasoning/reasoning_content as valid smoke signals.
Out of Scope Changes check	✅ Passed	All changes in both files are directly scoped to addressing #3341: test additions verify the max_tokens bump and reasoning-fallback logic, implementation changes only modify the smoke probe for reasoning-model compatibility.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/onboard.ts`:
- Around line 2579-2593: The parsing code can crash or mis-detect valid
responses: fix by guarding access to choices and by selecting a string-only
reasoning field. First, ensure you safely extract message by verifying
data.get("choices") is a non-empty list and that choices[0] is a dict before
calling .get (replace the current data.get("choices", [{}])[0] usage with an
explicit check for a list and len>0, then assign message = choices[0] if it's a
dict). Second, change the reasoning selection so you don't hide a string
reasoning_content behind a truthy non-string reasoning: explicitly test
message.get("reasoning_content") and message.get("reasoning") for str and
non-empty (prefer reasoning_content if it's a non-empty str, else reasoning if
it's a non-empty str) before treating it as a valid liveness signal; also ensure
content is only accepted if it's a str and non-empty.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d750ee9d-f538-4a9f-8895-63cc18cc13a6

📥 Commits

Reviewing files that changed from the base of the PR and between f66fb05 and 2e75cfe.

📒 Files selected for processing (2)

• src/lib/onboard.ts
• test/onboard.test.ts

[coderabbitai]

🧹 Nitpick comments (1)

test/onboard.test.ts (1)

706-722: ⚡ Quick win

Tighten parser-hardening assertions to match the stated fix scope

At Line 706 and Line 714, the new tests check key branches, but they don’t explicitly lock in the “non-empty string only” and “non-dict choice entry” guards described in the PR notes. Adding those assertions would prevent silent regressions in parser validation behavior.

Suggested test additions

   it("accepts a reasoning-only response as a valid smoke signal for thinking models (`#3341`)", () => {
     const script = buildCompatibleEndpointSandboxSmokeScript("Qwen/Qwen3.6-27B");

     assert.match(script, /message\.get\("reasoning"\)/);
     assert.match(script, /message\.get\("reasoning_content"\)/);
+    // Ensure fallback only accepts non-empty string reasoning payloads.
+    assert.match(script, /isinstance\([^,]+,\s*str\)/);
+    assert.match(script, /\.strip\(\)/);
     assert.match(script, /reasoning-only response/);
   });

   it("guards against empty or malformed choices arrays in the smoke parser (`#3341`)", () => {
     const script = buildCompatibleEndpointSandboxSmokeScript("Qwen/Qwen3.6-27B");

     assert.match(script, /not isinstance\(choices, list\) or not choices/);
+    // Ensure first choice shape is validated before message access.
+    assert.match(script, /not isinstance\([^,]+,\s*dict\)/);
     assert.doesNotMatch(script, /data\.get\("choices", \[\{\}\]\)\[0\]/);
   });

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/onboard.test.ts` around lines 706 - 722, Update the two smoke-parser
tests that use buildCompatibleEndpointSandboxSmokeScript to also assert the
tightened guards: for the reasoning-only branch assert the parser checks that
reasoning is a non-empty string (e.g., match a pattern like 'not
isinstance\\(reasoning, str\\) or not reasoning' or similar) and for the choices
guard assert there is an explicit check that each choice is a dict (e.g., match
'not all\\(isinstance\\(c, dict\\) for c in choices\\)' or 'not
isinstance\\(choice, dict\\)') so the tests lock in both the “non-empty string
only” and “non-dict choice entry” validations in the smoke parser used by
buildCompatibleEndpointSandboxSmokeScript.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/onboard.test.ts`:
- Around line 706-722: Update the two smoke-parser tests that use
buildCompatibleEndpointSandboxSmokeScript to also assert the tightened guards:
for the reasoning-only branch assert the parser checks that reasoning is a
non-empty string (e.g., match a pattern like 'not isinstance\\(reasoning, str\\)
or not reasoning' or similar) and for the choices guard assert there is an
explicit check that each choice is a dict (e.g., match 'not
all\\(isinstance\\(c, dict\\) for c in choices\\)' or 'not isinstance\\(choice,
dict\\)') so the tests lock in both the “non-empty string only” and “non-dict
choice entry” validations in the smoke parser used by
buildCompatibleEndpointSandboxSmokeScript.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5d4a7388-e7a1-42de-bf37-c233676eca29

📥 Commits

Reviewing files that changed from the base of the PR and between 2e75cfe and 3952f6c.

📒 Files selected for processing (2)

• src/lib/onboard.ts
• test/onboard.test.ts

[wscurran]

✨ Thanks for submitting this detailed PR to fix the onboard issue with thinking models like Qwen3.6. This change aims to improve the inference smoke probe by bumping max_tokens to 256 and treating non-empty message.reasoning or message.reasoning_content as a valid signal.

---

Related open issues:

• #3341 Cannot use thinking models like Qwen 3.6 27B - Nemoclaw validation request token length too small.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/onboard/compatible-endpoint-smoke.test.ts`:
- Around line 57-59: The negative assertion in the test currently checks for the
exact string '"max_tokens": 32,' which misses variants without the trailing
comma; update the assertion that uses the variable script in
compatible-endpoint-smoke.test.ts (the expect(script).not... line) to use a
regex-based negative match that rejects any occurrence of "max_tokens" followed
by optional whitespace, a colon, optional whitespace, and the value 32 (so it
fails for both '"max_tokens": 32,' and '"max_tokens": 32'), leaving the positive
check for 256 intact.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3422c17a-424f-4fc0-a50c-3b43e524700c

📥 Commits

Reviewing files that changed from the base of the PR and between 12c9526 and 9218417.

📒 Files selected for processing (2)

• src/lib/onboard/compatible-endpoint-smoke.test.ts
• src/lib/onboard/compatible-endpoint-smoke.ts

[latenighthackathon]

Closing as superseded by #3514, which landed 2026-05-13 and addresses the same root cause #3341.

#3514 takes a more thorough approach: it distinguishes route/config failures from model-budget failures, retries with a larger token budget (256 → 1024) when reasoning_content exhausts the initial budget, and adds an executable regression test for the MiniMax-shaped reasoning-only response. The compatible-endpoint smoke probe was also refactored in #3297 since this PR opened, so a rebase would have to be substantially reworked anyway.

Thanks for the review attention here. Cheers!