fix(inference): enable vLLM tool calls on generic-Linux Nemotron default - #6326
Conversation
The generic-Linux managed vLLM profile onboarded via `NEMOCLAW_EXPERIMENTAL=1 NEMOCLAW_PROVIDER=install-vllm` defaults to `nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8`, but the registry entry only pinned `--gpu-memory-utilization` and `--load-format`. Because the serve command carried neither `--enable-auto-tool-choice` nor `--tool-call-parser`, plain completions returned HTTP 200 but any agent request that sent `tool_choice: "auto"` failed HTTP 400 with vLLM's "'auto' tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set" — blocking every agent tool-call flow on the default local vLLM out of the box. The DGX Spark (`qwen3.6-35b-a3b-nvfp4` / `qwen3_xml`) and DGX Station (`deepseek-v4-flash` / `deepseek_v4`) defaults already pin their own tool-call parser; the generic-Linux Nemotron default was the outlier. Pin `--enable-auto-tool-choice` + `--tool-call-parser qwen3_coder` on the Nemotron-3-Nano-4B-FP8 registry entry, matching the vLLM launch example on the model card (https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8). This brings the generic-Linux managed vLLM default in line with the Spark and Station defaults and unblocks agent tool calls out of the box. Fixes #6314 Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>
📝 WalkthroughWalkthroughUpdates the Nemotron vLLM default model to include auto tool-choice flags and adds tests that verify the command output and Linux profile detection include them. ChangesvLLM tool-call flag fix
Estimated code review effort: 1 (Trivial) | ~5 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in the Show a code coverage summary of the most covered files.
TypeScript / code-coverage/cliThe overall coverage in the Show a code coverage summary of the most covered files.
Updated |
PR Review Advisor (Nemotron Ultra) — No blocking findingsMerge posture: No blocking advisor findings Action checklist
Findings index
Review findings by urgency: 0 required fixes, 2 items to resolve/justify, 1 in-scope improvement
|
PR Review Advisor — No blocking findingsMerge posture: No blocking advisor findings Action checklist
Test follow-ups to resolve or justifyIf these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.
This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision. |
E2E Advisor RecommendationRequired E2E: Dispatch hint: Full advisor summaryE2E Recommendation AdvisorBase: Required E2E
Optional E2E
New E2E recommendations
Dispatch hint
|
E2E Target RecommendationRequired E2E targets: None Full E2E target advisor summaryE2E Target AdvisorBase: Required E2E targets
Optional E2E targets
Relevant changed files
|
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
E2E Target Results — ✅ All requested jobs passedRun: 28838185730
|
E2E Target Results — ✅ All requested jobs passedRun: 28838310801
|
ericksoa
left a comment
There was a problem hiding this comment.
Approved at exact head 4f847e8f76373ab3d62386ea4902a0263a70f353 after current-main conflict audit and the nine-category sensitive-path review. I explicitly accept the unavailable generic-Linux discrete-GPU vLLM proof for this scoped registry fix: deterministic command/profile tests passed (47/47), the full inference suite passed (502 with 1 pre-existing skip), build/typecheck/check:diff and all 49 CI checks passed, inference-routing run 28838185730 passed, and advisor-required mcp-bridge plus both openclaw-inference-switch variants passed in run 28838310801. The change adds fixed argv tokens only and does not alter credentials, authentication, policy, or privilege boundaries.
<!-- markdownlint-disable MD041 --> ## Summary Prepares the user-facing documentation for NemoClaw v0.0.76 and closes the release-prep documentation gate. It adds the release highlights, documents the arm64 Local NIM warning and expanded image cleanup behavior, and fixes agent-specific command headings in generated guides. ## Changes - Add the v0.0.76 release-notes section and move the shared-gateway route containment entry out of the v0.0.74 history where it was incorrectly placed. - Document the advisory Linux arm64 Local NIM manifest warning in the canonical platform matrix and local-inference guidance. - Document that `gc` scans both gateway-built and locally prebuilt sandbox image repositories. - Keep OpenClaw and Hermes session headings out of the generated Deep Agents command guide. - Add a focused variant regression test for the agent-specific session headings. ### Source summary | Merged sources | Documentation coverage | | --- | --- | | [#6414](#6414), [#6418](#6418), [#6416](#6416), [#6344](#6344) | v0.0.76 release notes and the Deep Agents quickstart/inference routes | | [#6340](#6340) | v0.0.76 release notes and existing Deep Agents observability guidance | | [#6338](#6338), [#6378](#6378), [#6297](#6297) | v0.0.76 release notes and existing inference/troubleshooting guidance | | [#6362](#6362) | v0.0.76 release notes and existing lifecycle, command, and credential guidance | | [#6330](#6330), [#6307](#6307), [#6008](#6008) | v0.0.76 release notes and existing security, troubleshooting, and command guidance | | [#6382](#6382) | v0.0.76 release notes and existing MCP/command guidance | | [#6326](#6326), [#5868](#5868), [#5539](#5539) | v0.0.76 release notes, platform matrix, inference options, and local-inference guidance | | [#6396](#6396), [#6390](#6390), [#6007](#6007) | v0.0.76 release notes and existing messaging guidance | | [#5388](#5388), [#6249](#6249), [#6303](#6303), [#6306](#6306) | v0.0.76 release notes and command/lifecycle guidance | ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run --project integration test/generate-platform-docs.test.ts test/agent-variant-docs.test.ts test/sync-agent-variant-docs.test.ts` (3 files, 29 tests passed) - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — completed with 0 errors and 2 pre-existing Fern warnings - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added v0.0.76 release notes content, and removed an older conflicting bullet from the surrounding release history. * Expanded Local NVIDIA NIM guidance across inference/provider docs, including an advisory for Linux arm64 DGX Spark/DGX Station hosts when a matching `linux/arm64` image manifest is unavailable. * Updated the command reference for correct session-section rendering and clarified `gc` image cleanup sources. * **Tests** * Added coverage ensuring Deep Agents omits sessions headings while Hermes includes them. * **CI** * Refreshed Local NVIDIA NIM provider notes used in the platform matrix. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
…ult (NVIDIA#6326) <!-- markdownlint-disable MD041 --> ## Summary The generic-Linux managed-vLLM default, `nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8`, was launched without `--enable-auto-tool-choice` or a tool-call parser. Plain completions succeeded, but agent requests using `tool_choice: "auto"` failed HTTP 400. This PR pins `--tool-call-parser qwen3_coder`, matching NVIDIA's model-card launch recipe, and adds regression coverage at both the model-registry and generic-Linux profile layers. ## Related Issue Closes NVIDIA#6314. ## Reproduction and Analysis The reporter reproduced the failure on Ubuntu 24.04 x86_64 with an RTX 5070 (12 GB) using NemoClaw v0.0.74: ```bash NEMOCLAW_EXPERIMENTAL=1 NEMOCLAW_PROVIDER=install-vllm \ nemoclaw onboard --agent <agent> --name <name> --non-interactive --fresh --yes ``` Before this fix, `buildVllmServeCommand` produced a command ending with: ```text ... --gpu-memory-utilization 0.7 --load-format fastsafetensors ``` The reporter observed: ```text PLAIN: HTTP_CODE=200 TOOLCALL: HTTP_CODE=400 "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set ``` `GENERIC_LINUX_PROFILE.defaultModel` resolves to the `nemotron-3-nano-4b` registry entry. That entry was the outlier: the DGX Spark (`qwen3.6-35b-a3b-nvfp4` / `qwen3_xml`) and DGX Station (`deepseek-v4-flash` / `deepseek_v4`) defaults already pinned their own tool-call parsers. After this fix, the command ends with: ```text ... --gpu-memory-utilization 0.7 --load-format fastsafetensors \ --enable-auto-tool-choice --tool-call-parser qwen3_coder ``` The parser choice comes from the NVIDIA Nemotron-3-Nano-4B-FP8 model-card vLLM example and uses a parser already present in this registry. ## Changes - Add `--enable-auto-tool-choice --tool-call-parser qwen3_coder` to the Nemotron-3-Nano-4B FP8 registry entry. - Cover the generated serve command and the generic-Linux profile default with regression tests. - Require each tool-call switch to occur exactly once in the generated command. - Merge current `main` into the contributor branch without rewriting contributor history. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: this corrects the baked serve arguments for an existing managed-vLLM model and requires no new user action; current docs already describe the generic-Linux default, managed-vLLM limitations, and the need for model-appropriate tool-call flags. - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: maintainer review by @ericksoa on the final diff (2026-07-06) covered all nine security-review categories. The change adds fixed model-specific argv tokens, does not introduce user-controlled interpolation or alter credential/auth/policy boundaries, retains negative-path parser coverage, and passed gitleaks; no security findings. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable — `npm run check:diff` passed at `4f847e8f76373ab3d62386ea4902a0263a70f353`. - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run src/lib/inference/vllm-models.test.ts src/lib/inference/vllm.test.ts` (47/47 passed). - [x] Applicable broad gate passed — `npx vitest run src/lib/inference/` (502 passed, 1 pre-existing skip), `npm run typecheck:cli`, and `npm run build:cli` passed. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) ## Remaining Runtime Evidence Live end-to-end proof on an x86_64 generic-Linux NVIDIA GPU host is not included. The available fleet does not have that architecture/GPU combination; the DGX Spark host exercises a different profile that was already unaffected. The unit tests prove the exact command/profile composition, and the required `inference-routing` E2E is being run at the final head, but that E2E does not launch managed vLLM or validate this parser against the model. Before merge, a maintainer must either explicitly accept this hardware-evidence limitation or obtain a generic-Linux GPU run showing the fixed command and a successful `tool_choice: "auto"` request. ## AI Disclosure - [x] AI-assisted — contributor tool: Claude Code; maintainer salvage tool: Codex --- Signed-off-by: Yanyun Liao <yanyunl@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Updated vLLM settings for the Nemotron-3-Nano-4B FP8 model to enable automatic tool choice with the correct tool-call parser configuration. * Fixed Linux/NVIDIA vLLM profile detection to include the expected tool-call parser settings. * **Tests** * Added regression coverage to ensure the tool-call related flags are included correctly going forward. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Yanyun Liao <yanyunl@nvidia.com> Signed-off-by: Aaron Erickson <aerickson@nvidia.com> Co-authored-by: Aaron Erickson <aerickson@nvidia.com>
<!-- markdownlint-disable MD041 --> ## Summary Prepares the user-facing documentation for NemoClaw v0.0.76 and closes the release-prep documentation gate. It adds the release highlights, documents the arm64 Local NIM warning and expanded image cleanup behavior, and fixes agent-specific command headings in generated guides. ## Changes - Add the v0.0.76 release-notes section and move the shared-gateway route containment entry out of the v0.0.74 history where it was incorrectly placed. - Document the advisory Linux arm64 Local NIM manifest warning in the canonical platform matrix and local-inference guidance. - Document that `gc` scans both gateway-built and locally prebuilt sandbox image repositories. - Keep OpenClaw and Hermes session headings out of the generated Deep Agents command guide. - Add a focused variant regression test for the agent-specific session headings. ### Source summary | Merged sources | Documentation coverage | | --- | --- | | [NVIDIA#6414](NVIDIA#6414), [NVIDIA#6418](NVIDIA#6418), [NVIDIA#6416](NVIDIA#6416), [NVIDIA#6344](NVIDIA#6344) | v0.0.76 release notes and the Deep Agents quickstart/inference routes | | [NVIDIA#6340](NVIDIA#6340) | v0.0.76 release notes and existing Deep Agents observability guidance | | [NVIDIA#6338](NVIDIA#6338), [NVIDIA#6378](NVIDIA#6378), [NVIDIA#6297](NVIDIA#6297) | v0.0.76 release notes and existing inference/troubleshooting guidance | | [NVIDIA#6362](NVIDIA#6362) | v0.0.76 release notes and existing lifecycle, command, and credential guidance | | [NVIDIA#6330](NVIDIA#6330), [NVIDIA#6307](NVIDIA#6307), [NVIDIA#6008](NVIDIA#6008) | v0.0.76 release notes and existing security, troubleshooting, and command guidance | | [NVIDIA#6382](NVIDIA#6382) | v0.0.76 release notes and existing MCP/command guidance | | [NVIDIA#6326](NVIDIA#6326), [NVIDIA#5868](NVIDIA#5868), [NVIDIA#5539](NVIDIA#5539) | v0.0.76 release notes, platform matrix, inference options, and local-inference guidance | | [NVIDIA#6396](NVIDIA#6396), [NVIDIA#6390](NVIDIA#6390), [NVIDIA#6007](NVIDIA#6007) | v0.0.76 release notes and existing messaging guidance | | [NVIDIA#5388](NVIDIA#5388), [NVIDIA#6249](NVIDIA#6249), [NVIDIA#6303](NVIDIA#6303), [NVIDIA#6306](NVIDIA#6306) | v0.0.76 release notes and command/lifecycle guidance | ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run --project integration test/generate-platform-docs.test.ts test/agent-variant-docs.test.ts test/sync-agent-variant-docs.test.ts` (3 files, 29 tests passed) - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — completed with 0 errors and 2 pre-existing Fern warnings - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added v0.0.76 release notes content, and removed an older conflicting bullet from the surrounding release history. * Expanded Local NVIDIA NIM guidance across inference/provider docs, including an advisory for Linux arm64 DGX Spark/DGX Station hosts when a matching `linux/arm64` image manifest is unavailable. * Updated the command reference for correct session-section rendering and clarified `gc` image cleanup sources. * **Tests** * Added coverage ensuring Deep Agents omits sessions headings while Hermes includes them. * **CI** * Refreshed Local NVIDIA NIM provider notes used in the platform matrix. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Summary
The generic-Linux managed-vLLM default,
nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8, was launched without--enable-auto-tool-choiceor a tool-call parser. Plain completions succeeded, but agent requests usingtool_choice: "auto"failed HTTP 400. This PR pins--tool-call-parser qwen3_coder, matching NVIDIA's model-card launch recipe, and adds regression coverage at both the model-registry and generic-Linux profile layers.Related Issue
Closes #6314.
Reproduction and Analysis
The reporter reproduced the failure on Ubuntu 24.04 x86_64 with an RTX 5070 (12 GB) using NemoClaw v0.0.74:
Before this fix,
buildVllmServeCommandproduced a command ending with:The reporter observed:
GENERIC_LINUX_PROFILE.defaultModelresolves to thenemotron-3-nano-4bregistry entry. That entry was the outlier: the DGX Spark (qwen3.6-35b-a3b-nvfp4/qwen3_xml) and DGX Station (deepseek-v4-flash/deepseek_v4) defaults already pinned their own tool-call parsers.After this fix, the command ends with:
The parser choice comes from the NVIDIA Nemotron-3-Nano-4B-FP8 model-card vLLM example and uses a parser already present in this registry.
Changes
--enable-auto-tool-choice --tool-call-parser qwen3_coderto the Nemotron-3-Nano-4B FP8 registry entry.maininto the contributor branch without rewriting contributor history.Type of Change
Quality Gates
Verification
Verifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailable —npm run check:diffpassed at4f847e8f76373ab3d62386ea4902a0263a70f353.npx vitest run src/lib/inference/vllm-models.test.ts src/lib/inference/vllm.test.ts(47/47 passed).npx vitest run src/lib/inference/(502 passed, 1 pre-existing skip),npm run typecheck:cli, andnpm run build:clipassed.npm run docsbuilds without warnings (doc changes only)Remaining Runtime Evidence
Live end-to-end proof on an x86_64 generic-Linux NVIDIA GPU host is not included. The available fleet does not have that architecture/GPU combination; the DGX Spark host exercises a different profile that was already unaffected. The unit tests prove the exact command/profile composition, and the required
inference-routingE2E is being run at the final head, but that E2E does not launch managed vLLM or validate this parser against the model. Before merge, a maintainer must either explicitly accept this hardware-evidence limitation or obtain a generic-Linux GPU run showing the fixed command and a successfultool_choice: "auto"request.AI Disclosure
Signed-off-by: Yanyun Liao yanyunl@nvidia.com
Summary by CodeRabbit