fix(inference): request a reply budget hosted endpoints accept

[nvshaxie]

Summary

Inference probes hard-coded a bounded reply budget of 8 tokens. That budget is a value the endpoint validates, not just a ceiling NemoClaw applies, so hosted endpoints that enforce a higher floor rejected the probe with HTTP 400 even though model discovery succeeded and normal inference worked. A valid endpoint, model, and credential combination therefore failed onboarding, health, and rebuild preflight. The probes now share one minimum reply budget of 16 and stay tightly bounded.

Related Issue

Refs: #7939

Changes

• Add MIN_PROBE_REPLY_TOKENS (16) to src/lib/inference/max-tokens-field.ts, the module both probe builders already share for reply-budget decisions.
• Use it for the Chat Completions probe payload and its Kimi K2.6 branch (openai-probe-models.ts), the provider health probe (health.ts), and all three rebuild-preflight routes — Chat Completions, Responses, and Anthropic Messages (rebuild-inference-preflight.ts).
• Cover every preflight route and both Chat Completions budget fields with regression tests, so this cannot regress into the gpt-5.4 on a custom OpenAI-compatible provider fails: chat/completions rejects max_tokens (needs max_completion_tokens) #6642 field-name contract.

This is a direct value change. It adds no configuration, fallback, compatibility, or extension surface: the floor is one shared constant with three consumers, protected by the tests above. The value matches the floor the Anthropic Messages probe (probe-anthropic.ts) already used, so it introduces no new number to the codebase. The DeepSeek V4 Pro branch keeps its larger 8192 validation budget.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: no command, flag, JSON field, text output, exit behavior, or configuration surface changes. docs/inference/choose-compatible-inference-api.mdx documents the reply-budget field name selection (gpt-5.4 on a custom OpenAI-compatible provider fails: chat/completions rejects max_tokens (needs max_completion_tokens) #6642), which is unchanged, and docs/inference/configure-model-limits.mdx documents user-configured sandbox model limits, which are a separate concept from this internal probe budget. No page states the probe budget value.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent Codex Desktop security review completed for 56f8dadb4. All nine categories passed with no findings: secrets and credentials; input validation and data sanitization; authentication and authorization; dependencies and third-party libraries; error handling and logging; cryptography and data protection; configuration and security headers; security testing; and system security. The change keeps credentials out of commands and output, preserves response redaction and SSRF controls, and keeps the probe bounded at 16 tokens.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: no-docs-needed
• Evidence: Internal onboarding, inference-health, and rebuild probes now request 16 reply tokens instead of 8. Existing docs do not specify this implementation value; commands, configuration, output text, and operator procedures are unchanged. Reviewed all changed comments and test titles.
• Agent: Codex Desktop

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub — all four PR commits, including the two maintainer refresh commits through 56f8dadb4, are verified.
• Normal pre-commit, commit-msg, and pre-push hooks passed. The refreshed pre-push hook completed during publication of 56f8dadb4.
• Targeted behavior tests passed for contributor commit 30904bc49 after npm run build:cli:
  • npx vitest run --project cli src/lib/inference/onboard-probes.test.ts src/lib/inference/health.test.ts src/lib/actions/sandbox/rebuild-inference-preflight.test.ts → 3 files, 85 passed
  • npx vitest run --project e2e-support → 172 files, 1840 passed, 17 skipped
  • npm run typecheck:cli → clean
  • Negative check: reverting MIN_PROBE_REPLY_TOKENS to 8 fails all five new (#7939) cases, so the added tests genuinely pin the fixed behavior.
• GitHub CI passes for refreshed commit 56f8dadb4 — pending publication.
• Applicable broad gate passed — not applicable; this is a scoped inference change, not a runtime/test-harness or repo-wide validation change.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

Verification boundary

I could not reproduce the reporter's live HTTP 400 end to end: this host has no OpenShell sandbox, and the failing endpoint is an NVIDIA hosted route the report deliberately does not name. The fix is verified against the source contract the report cites — all three quoted call sites, plus the Responses and Anthropic routes the report did not enumerate — and by mock-level regression tests. Live confirmation on the reporter's endpoint with openai/openai/gpt-5.6-sol is still worth having before this is treated as field-proven.

Unrelated observation: src/lib/inference/nim.test.ts > detectGpu > returns object or null fails locally with a 5s test timeout because nvidia-smi takes ~10s on this multi-A100 host. It fails nondeterministically on unmodified main too and shares no import path with these files.

---

Signed-off-by: Shawn Xie shaxie@nvidia.com

Summary by CodeRabbit

• Bug Fixes

  • Improved inference health checks and preflight validation by consistently allocating a minimum 16-token reply budget.
  • Updated support for multiple request formats, including Responses API, chat completions, and Anthropic requests.
  • Improved probe reliability across supported model families, including reasoning and streaming models.

• Tests

  • Added coverage to verify token-budget fields and minimum values across inference routes and model types.

[coderabbitai]

📝 Walkthrough

Walkthrough

Probe requests now use a shared minimum reply budget of 16 tokens across OpenAI, Anthropic, Responses, chat-completions, and streaming formats. Tests update expected payloads and validate route-specific token fields across supported models.

Changes

Inference probe budget

Layer / File(s)	Summary
Shared budget contract src/lib/inference/max-tokens-field.ts, src/lib/inference/openai-probe-models.ts	Exports MIN_PROBE_REPLY_TOKENS = 16 and applies it to default and Kimi probe payloads.
Probe payload wiring src/lib/inference/health.ts, src/lib/actions/sandbox/rebuild-inference-preflight.ts	Uses the shared minimum for health and sandbox preflight requests across supported route formats.
Budget validation src/lib/inference/*.test.ts, src/lib/actions/sandbox/rebuild-inference-preflight.test.ts	Updates expected budgets to 16 and adds cross-route and cross-model minimum-budget assertions.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• NVIDIA/NemoClaw#5400: Modifies onboarding and inference probe payload token budgets and related tests.

Suggested labels: area: onboarding

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the inference fix to request a reply budget accepted by hosted endpoints.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/7939-probe-reply-budget-minimum

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 56f8dad in the fix/7939-probe-reply... branch remains at 96%, unchanged from commit b724932 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 56f8dad in the fix/7939-probe-reply... branch remains at 81%, unchanged from commit 643a4ab in the main branch.

Show a code coverage summary of the most impacted files.

File	main 643a4ab	fix/7939-probe-reply... 56f8dad	+/-
src/lib/onboard...shboard-port.ts	96%	90%	-6%
src/lib/platform.ts	89%	84%	-5%
src/lib/inferen...tokens-field.ts	100%	100%	0%
src/lib/messagi...nnels/policy.ts	100%	100%	0%
src/lib/sandbox...rce-identity.ts	88%	88%	0%
src/lib/state/config-io.ts	93%	93%	0%
src/lib/state/g...way-registry.ts	97%	97%	0%
src/lib/tunnel/services.ts	76%	76%	0%
src/lib/inference/health.ts	92%	93%	+1%
src/lib/actions...ce-preflight.ts	50%	78%	+28%

---

_{Updated August 04, 2026 06:36 UTC}

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · medium confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized terminology decisions differ; normalized E2E selections differ; severity counts match.

3 additional E2E selections from the second opinion

Advisory only. The primary lane did not select these E2E jobs or targets.

• cloud-onboard: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
• cloud-inference: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
• kimi-inference-compat: The completed second-opinion lane identified E2E coverage that the primary lane omitted.

Second-opinion terminology and E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate.

1 semantic terminology decision

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

• established — reply budget at src/lib/inference/max-tokens-field.ts:5: Keep `reply budget`. The changed text correctly states that probes use a minimum reply budget.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: inference-routing, network-policy, onboard-repair, onboard-resume, rebuild-openclaw, state-backup-restore

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[apurvvkumaria]

Reviewed exact head 30904bc. The shared minimum probe reply budget is applied consistently across chat completions, responses, Anthropic, health, onboarding, and rebuild paths, with focused route and field coverage. I found no blocking correctness, security, compatibility, or regression issue; current CI is green.

[apurvvkumaria]

Re-reviewed current head b20f99b after the merge from main. The only combined-resolution area is src/lib/inference/onboard-probes.test.ts: it preserves this PRs 16-token hosted-endpoint floor while also retaining mains per-request curl argument capture and validation-timeout assertions. The harness counter now records arguments before parsing without changing response selection, and environment cleanup remains in finally. I found no new blocking defect. Exact-head CI is running; the image-build failures are the same inherited MCP runtime npm-audit findings and are unrelated to this inference/test merge.