Skip to content

fix(inference): enable vLLM tool calls on generic-Linux Nemotron default - #6326

Merged
cv merged 3 commits into
mainfrom
fix/vllm-linux-nemotron-tool-call-parser-6314
Jul 7, 2026
Merged

fix(inference): enable vLLM tool calls on generic-Linux Nemotron default#6326
cv merged 3 commits into
mainfrom
fix/vllm-linux-nemotron-tool-call-parser-6314

Conversation

@yanyunl1991

@yanyunl1991 yanyunl1991 commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

The generic-Linux managed-vLLM default, nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8, was launched without --enable-auto-tool-choice or a tool-call parser. Plain completions succeeded, but agent requests using tool_choice: "auto" failed HTTP 400. This PR pins --tool-call-parser qwen3_coder, matching NVIDIA's model-card launch recipe, and adds regression coverage at both the model-registry and generic-Linux profile layers.

Related Issue

Closes #6314.

Reproduction and Analysis

The reporter reproduced the failure on Ubuntu 24.04 x86_64 with an RTX 5070 (12 GB) using NemoClaw v0.0.74:

NEMOCLAW_EXPERIMENTAL=1 NEMOCLAW_PROVIDER=install-vllm \
  nemoclaw onboard --agent <agent> --name <name> --non-interactive --fresh --yes

Before this fix, buildVllmServeCommand produced a command ending with:

... --gpu-memory-utilization 0.7 --load-format fastsafetensors

The reporter observed:

PLAIN:    HTTP_CODE=200
TOOLCALL: HTTP_CODE=400
"auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set

GENERIC_LINUX_PROFILE.defaultModel resolves to the nemotron-3-nano-4b registry entry. That entry was the outlier: the DGX Spark (qwen3.6-35b-a3b-nvfp4 / qwen3_xml) and DGX Station (deepseek-v4-flash / deepseek_v4) defaults already pinned their own tool-call parsers.

After this fix, the command ends with:

... --gpu-memory-utilization 0.7 --load-format fastsafetensors \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

The parser choice comes from the NVIDIA Nemotron-3-Nano-4B-FP8 model-card vLLM example and uses a parser already present in this registry.

Changes

  • Add --enable-auto-tool-choice --tool-call-parser qwen3_coder to the Nemotron-3-Nano-4B FP8 registry entry.
  • Cover the generated serve command and the generic-Linux profile default with regression tests.
  • Require each tool-call switch to occur exactly once in the generated command.
  • Merge current main into the contributor branch without rewriting contributor history.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: this corrects the baked serve arguments for an existing managed-vLLM model and requires no new user action; current docs already describe the generic-Linux default, managed-vLLM limitations, and the need for model-appropriate tool-call flags.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: maintainer review by @ericksoa on the final diff (2026-07-06) covered all nine security-review categories. The change adds fixed model-specific argv tokens, does not introduce user-controlled interpolation or alter credential/auth/policy boundaries, retains negative-path parser coverage, and passed gitleaks; no security findings.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable — npm run check:diff passed at 4f847e8f76373ab3d62386ea4902a0263a70f353.
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run src/lib/inference/vllm-models.test.ts src/lib/inference/vllm.test.ts (47/47 passed).
  • Applicable broad gate passed — npx vitest run src/lib/inference/ (502 passed, 1 pre-existing skip), npm run typecheck:cli, and npm run build:cli passed.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Remaining Runtime Evidence

Live end-to-end proof on an x86_64 generic-Linux NVIDIA GPU host is not included. The available fleet does not have that architecture/GPU combination; the DGX Spark host exercises a different profile that was already unaffected. The unit tests prove the exact command/profile composition, and the required inference-routing E2E is being run at the final head, but that E2E does not launch managed vLLM or validate this parser against the model. Before merge, a maintainer must either explicitly accept this hardware-evidence limitation or obtain a generic-Linux GPU run showing the fixed command and a successful tool_choice: "auto" request.

AI Disclosure

  • AI-assisted — contributor tool: Claude Code; maintainer salvage tool: Codex

Signed-off-by: Yanyun Liao yanyunl@nvidia.com

Summary by CodeRabbit

  • Bug Fixes
    • Updated vLLM settings for the Nemotron-3-Nano-4B FP8 model to enable automatic tool choice with the correct tool-call parser configuration.
    • Fixed Linux/NVIDIA vLLM profile detection to include the expected tool-call parser settings.
  • Tests
    • Added regression coverage to ensure the tool-call related flags are included correctly going forward.

The generic-Linux managed vLLM profile onboarded via
`NEMOCLAW_EXPERIMENTAL=1 NEMOCLAW_PROVIDER=install-vllm` defaults to
`nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8`, but the registry entry only
pinned `--gpu-memory-utilization` and `--load-format`. Because the
serve command carried neither `--enable-auto-tool-choice` nor
`--tool-call-parser`, plain completions returned HTTP 200 but any
agent request that sent `tool_choice: "auto"` failed HTTP 400 with
vLLM's "'auto' tool choice requires --enable-auto-tool-choice and
--tool-call-parser to be set" — blocking every agent tool-call flow
on the default local vLLM out of the box.

The DGX Spark (`qwen3.6-35b-a3b-nvfp4` / `qwen3_xml`) and DGX Station
(`deepseek-v4-flash` / `deepseek_v4`) defaults already pin their own
tool-call parser; the generic-Linux Nemotron default was the outlier.

Pin `--enable-auto-tool-choice` + `--tool-call-parser qwen3_coder` on
the Nemotron-3-Nano-4B-FP8 registry entry, matching the vLLM launch
example on the model card
(https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8). This
brings the generic-Linux managed vLLM default in line with the Spark
and Station defaults and unblocks agent tool calls out of the box.

Fixes #6314

Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Updates the Nemotron vLLM default model to include auto tool-choice flags and adds tests that verify the command output and Linux profile detection include them.

Changes

vLLM tool-call flag fix

Layer / File(s) Summary
Add tool-call flags to model registry
src/lib/inference/vllm-models.ts
modelArgs for the Nemotron-3-Nano-4B-FP8 entry now includes --enable-auto-tool-choice and --tool-call-parser qwen3_coder, alongside the existing GPU memory utilization and load-format flags, with expanded comments.
Regression tests for tool-call flags
src/lib/inference/vllm-models.test.ts, src/lib/inference/vllm.test.ts
New tests assert buildVllmServeCommand output and the Linux default model's modelArgs include the tool-call flags exactly once.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: cv, jyaunches, ericksoa

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The registry update and regression tests satisfy #6314 by adding the required tool-call flags for the generic-Linux default.
Out of Scope Changes check ✅ Passed The diff stays focused on the reported vLLM default-model fix and supporting tests, with no clear unrelated additions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: enabling vLLM tool calls for the generic-Linux Nemotron default.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/vllm-linux-nemotron-tool-call-parser-6314

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the fix/vllm-linux-nemot... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main fix/vllm-linux-nemot... 4f847e8 +/-
nemoclaw/src/se...cret-scanner.ts 100%
nemoclaw/src/commands/slash.ts 100%
nemoclaw/src/li...bprocess-env.ts 100%
nemoclaw/src/bl...eprint/state.ts 98%
nemoclaw/src/onboard/config.ts 98%
nemoclaw/src/bl...int/snapshot.ts 97%
nemoclaw/src/bl...print/runner.ts 95%
nemoclaw/src/co...ration-state.ts 94%
nemoclaw/src/bl...ate-networks.ts 94%
nemoclaw/src/index.ts 94%

TypeScript / code-coverage/cli

The overall coverage in the fix/vllm-linux-nemot... branch is 73%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main fix/vllm-linux-nemot... 4f847e8 +/-
src/lib/shields...nsition-lock.ts 85%
src/lib/onboard/preflight.ts 83%
src/lib/actions...all/run-plan.ts 81%
src/lib/state/o...oard-session.ts 81%
src/lib/actions...licy-channel.ts 79%
src/lib/state/sandbox.ts 75%
src/lib/onboard...er-gpu-patch.ts 69%
src/lib/shields/index.ts 68%
src/lib/policy/index.ts 66%
src/lib/onboard.ts 28%

Updated July 07, 2026 03:03 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor (Nemotron Ultra) — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: Resolve or justify PRA-1: Container logs printed on failure bypass runner redaction.
Open items: 0 required · 2 warnings · 1 suggestion · 2 test follow-ups
Since last review: 0 prior items resolved · 1 still applies · 2 new items found

Action checklist

  • PRA-1 Resolve or justify: Container logs printed on failure bypass runner redaction in src/lib/inference/vllm.ts:369
  • PRA-2 Resolve or justify: Container runs with elevated privileges and --trust-remote-code in src/lib/inference/vllm.ts:390
  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Consider test for duplicate flag injection via extra args
  • PRA-3 In-scope improvement: Consider test for duplicate flag injection via extra args in src/lib/inference/vllm-models.test.ts:140

Findings index

ID Severity Category Location Required action
PRA-1 Resolve/justify security src/lib/inference/vllm.ts:369 Consider redacting docker logs output or suppressing env vars in container logs. Verify HF_TOKEN not leaked via `docker logs` in failure path.
PRA-2 Resolve/justify security src/lib/inference/vllm.ts:390 Confirm localhost-only binding is enforced. Document --trust-remote-code risk and registry trust model. Not a PR-specific issue — pre-existing architecture.
PRA-3 Improvement tests src/lib/inference/vllm-models.test.ts:140 Add test asserting extra args validation rejects known vLLM flags, or document that duplicates cause vLLM error. Current-PR action recommended since test file is already modified.
Review findings by urgency: 0 required fixes, 2 items to resolve/justify, 1 in-scope improvement

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

PRA-1 Resolve/justify — Container logs printed on failure bypass runner redaction

  • Location: src/lib/inference/vllm.ts:369
  • Category: security
  • Problem: printContainerLogTail calls dockerCapture directly and writes raw output to stderr, bypassing runner's writeRedactedResult. On failure, vLLM server logs (which may contain request data) are printed unredacted. HF token not in docker argv (key-only -e) but container env has it; docker logs may not show env but vLLM request logs could contain sensitive data.
  • Impact: Potential credential or sensitive data leakage in failure logs when vLLM container fails to start or crashes.
  • Recommended action: Consider redacting docker logs output or suppressing env vars in container logs. Verify HF_TOKEN not leaked via `docker logs` in failure path.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Check vllm.ts:369-380 for printContainerLogTail; vllm.ts:437-445 for waitForVllmReady failure path; runner.ts writeRedactedResult vs dockerCapture direct use
  • Missing regression test: Add test that verifies docker logs output on failure does not contain HF_TOKEN pattern (hf_...)
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Check vllm.ts:369-380 for printContainerLogTail; vllm.ts:437-445 for waitForVllmReady failure path; runner.ts writeRedactedResult vs dockerCapture direct use.
  • Evidence: Runner redacts stdout/stderr via writeRedactedResult using redact() which covers hf_ tokens. However, printContainerLogTail calls dockerCapture directly and writes raw output to stderr — this bypasses runner redaction.

PRA-2 Resolve/justify — Container runs with elevated privileges and --trust-remote-code

  • Location: src/lib/inference/vllm.ts:390
  • Category: security
  • Problem: vLLM container binds port 8000 to localhost only. Container runs with --gpus all, --ipc=host, volume mounts for HF cache, --restart unless-stopped. --trust-remote-code flag enables arbitrary code execution from model repos — mitigated by code-defined registry allowlist. No CSP/CORS configured (internal localhost service).
  • Impact: Pre-existing architecture risk: if registry allowlist is compromised or model repo is malicious, --trust-remote-code allows arbitrary code execution in container with GPU access and host IPC.
  • Recommended action: Confirm localhost-only binding is enforced. Document --trust-remote-code risk and registry trust model. Not a PR-specific issue — pre-existing architecture.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Check vllm.ts:390-395 for buildVllmRunCommand port binding; vllm.ts:53-58 for VLLM_PORT=8000; vllm-models.ts:350-355 for SHARED_VLLM_ARGS with --trust-remote-code
  • Missing regression test: Add test verifying -p 8000:8000 (not -p 0.0.0.0:8000:8000) in buildVllmRunCommand output
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Check vllm.ts:390-395 for buildVllmRunCommand port binding; vllm.ts:53-58 for VLLM_PORT=8000; vllm-models.ts:350-355 for SHARED_VLLM_ARGS with --trust-remote-code.
  • Evidence: Port 8000 bound to localhost only (127.0.0.1:8000). Container runs with --gpus all, --ipc=host, volume mounts, --restart unless-stopped. --trust-remote-code required by vLLM for custom models; mitigated by code-defined registry allowlist.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

PRA-3 Improvement — Consider test for duplicate flag injection via extra args

  • Location: src/lib/inference/vllm-models.test.ts:140
  • Category: tests
  • Problem: User could pass --enable-auto-tool-choice in NEMOCLAW_VLLM_EXTRA_ARGS_JSON causing duplicate in final command. vLLM would error (fail-fast), but explicit test would document behavior and prevent future regression if vLLM changes to allow duplicates.
  • Impact: Low — vLLM errors on duplicate flags, but behavior not explicitly tested.
  • Suggested action: Add test asserting extra args validation rejects known vLLM flags, or document that duplicates cause vLLM error. Current-PR action recommended since test file is already modified.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Check vllm-models.test.ts:27-65 for extra args parsing tests; vllm-models.ts:323-360 for parseVllmExtraServeArgs
  • Missing regression test: Test that NEMOCLAW_VLLM_EXTRA_ARGS_JSON containing --enable-auto-tool-choice produces duplicate flag in final command (current behavior) or is rejected (proposed enhancement)
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Extra args append after modelArgs; no validation against known vLLM flags. vLLM would error on duplicate --enable-auto-tool-choice.
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — End-to-end installVllm + agent tool-call flow on generic Linux (non-Spark/Station) with Nemotron-3-Nano-4B-FP8, verifying tool_choice:auto succeeds. Unit tests cover exact code path (buildVllmServeCommand composition + detectVllmProfile platform resolution) with 47/47 tests passing in changed files and 492/492 across inference scope. Runtime validation on generic-Linux x86_64 GPU host blocked by hardware availability (fleet lacks this arch/GPU combination), not test design.
  • PRA-T2 Consider test for duplicate flag injection via extra args — Add test asserting extra args validation rejects known vLLM flags, or document that duplicates cause vLLM error. Current-PR action recommended since test file is already modified.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Resolve/justify — Container logs printed on failure bypass runner redaction

  • Location: src/lib/inference/vllm.ts:369
  • Category: security
  • Problem: printContainerLogTail calls dockerCapture directly and writes raw output to stderr, bypassing runner's writeRedactedResult. On failure, vLLM server logs (which may contain request data) are printed unredacted. HF token not in docker argv (key-only -e) but container env has it; docker logs may not show env but vLLM request logs could contain sensitive data.
  • Impact: Potential credential or sensitive data leakage in failure logs when vLLM container fails to start or crashes.
  • Recommended action: Consider redacting docker logs output or suppressing env vars in container logs. Verify HF_TOKEN not leaked via `docker logs` in failure path.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Check vllm.ts:369-380 for printContainerLogTail; vllm.ts:437-445 for waitForVllmReady failure path; runner.ts writeRedactedResult vs dockerCapture direct use
  • Missing regression test: Add test that verifies docker logs output on failure does not contain HF_TOKEN pattern (hf_...)
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Check vllm.ts:369-380 for printContainerLogTail; vllm.ts:437-445 for waitForVllmReady failure path; runner.ts writeRedactedResult vs dockerCapture direct use.
  • Evidence: Runner redacts stdout/stderr via writeRedactedResult using redact() which covers hf_ tokens. However, printContainerLogTail calls dockerCapture directly and writes raw output to stderr — this bypasses runner redaction.

PRA-2 Resolve/justify — Container runs with elevated privileges and --trust-remote-code

  • Location: src/lib/inference/vllm.ts:390
  • Category: security
  • Problem: vLLM container binds port 8000 to localhost only. Container runs with --gpus all, --ipc=host, volume mounts for HF cache, --restart unless-stopped. --trust-remote-code flag enables arbitrary code execution from model repos — mitigated by code-defined registry allowlist. No CSP/CORS configured (internal localhost service).
  • Impact: Pre-existing architecture risk: if registry allowlist is compromised or model repo is malicious, --trust-remote-code allows arbitrary code execution in container with GPU access and host IPC.
  • Recommended action: Confirm localhost-only binding is enforced. Document --trust-remote-code risk and registry trust model. Not a PR-specific issue — pre-existing architecture.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Check vllm.ts:390-395 for buildVllmRunCommand port binding; vllm.ts:53-58 for VLLM_PORT=8000; vllm-models.ts:350-355 for SHARED_VLLM_ARGS with --trust-remote-code
  • Missing regression test: Add test verifying -p 8000:8000 (not -p 0.0.0.0:8000:8000) in buildVllmRunCommand output
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Check vllm.ts:390-395 for buildVllmRunCommand port binding; vllm.ts:53-58 for VLLM_PORT=8000; vllm-models.ts:350-355 for SHARED_VLLM_ARGS with --trust-remote-code.
  • Evidence: Port 8000 bound to localhost only (127.0.0.1:8000). Container runs with --gpus all, --ipc=host, volume mounts, --restart unless-stopped. --trust-remote-code required by vLLM for custom models; mitigated by code-defined registry allowlist.

PRA-3 Improvement — Consider test for duplicate flag injection via extra args

  • Location: src/lib/inference/vllm-models.test.ts:140
  • Category: tests
  • Problem: User could pass --enable-auto-tool-choice in NEMOCLAW_VLLM_EXTRA_ARGS_JSON causing duplicate in final command. vLLM would error (fail-fast), but explicit test would document behavior and prevent future regression if vLLM changes to allow duplicates.
  • Impact: Low — vLLM errors on duplicate flags, but behavior not explicitly tested.
  • Suggested action: Add test asserting extra args validation rejects known vLLM flags, or document that duplicates cause vLLM error. Current-PR action recommended since test file is already modified.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Check vllm-models.test.ts:27-65 for extra args parsing tests; vllm-models.ts:323-360 for parseVllmExtraServeArgs
  • Missing regression test: Test that NEMOCLAW_VLLM_EXTRA_ARGS_JSON containing --enable-auto-tool-choice produces duplicate flag in final command (current behavior) or is rejected (proposed enhancement)
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Extra args append after modelArgs; no validation against known vLLM flags. vLLM would error on duplicate --enable-auto-tool-choice.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: Add or justify PRA-T1 and any related test follow-ups.
Open items: 0 required · 0 warnings · 0 suggestions · 2 test follow-ups
Since last review: 0 prior items resolved · 2 still apply · 0 new items found

Action checklist

  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — On a generic-Linux NVIDIA GPU host, onboard managed vLLM with the default Nemotron model and POST a chat completion containing `tools` plus `tool_choice:"auto"`; verify the response is not the reported HTTP 400 and includes structured tool-call handling.. The changed static registry/profile behavior is well covered by unit tests, but the end-user symptom depends on a generic-Linux NVIDIA GPU runtime path and an actual vLLM server accepting `tool_choice: "auto"` requests.
  • PRA-T2 Runtime validation — After onboarding on a generic-Linux NVIDIA GPU host, inspect the running vLLM container or process command for `vllm serve nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8` and verify `--enable-auto-tool-choice --tool-call-parser qwen3_coder` appears exactly once.. The changed static registry/profile behavior is well covered by unit tests, but the end-user symptom depends on a generic-Linux NVIDIA GPU runtime path and an actual vLLM server accepting `tool_choice: "auto"` requests.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: mcp-bridge, openclaw-inference-switch
Optional E2E: gpu-e2e

Dispatch hint: mcp-bridge,openclaw-inference-switch

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • mcp-bridge (high): Closest existing live E2E for the affected user-flow class: it exercises real assistant MCP/tool-call flows across OpenClaw/Hermes/DeepAgents. The vLLM change exists to make tool_choice:auto agent requests work, so this should be merge-blocking even though it does not directly launch managed vLLM.
  • openclaw-inference-switch (high): Validates the live OpenClaw inference route/config/registry path and inference.local behavior after provider routing changes. This is adjacent coverage for inference-routing regressions from changes under src/lib/inference.

Optional E2E

  • gpu-e2e (very high): Optional adjacent confidence for local GPU-backed inference.local onboarding and sandbox inference behavior. It uses Ollama rather than vLLM, so it is not direct coverage for the changed vLLM serve flags.

New E2E recommendations

  • managed-vllm-generic-linux-tool-choice (high): No existing E2E appears to launch the managed vLLM generic-Linux default and send an agent/tool-call request with tool_choice:auto through inference.local. The current required jobs cover adjacent live tool-call and inference-routing boundaries but would not catch a missing vLLM serve flag.
    • Suggested test: Add a GPU-backed managed-vLLM E2E that onboards with NEMOCLAW_PROVIDER=vllm-local or the install-vllm path on generic Linux, waits for the Nemotron-3-Nano-4B FP8 server, then verifies an OpenClaw agent/tool-call request using tool_choice:auto succeeds instead of returning vLLM HTTP 400.

Dispatch hint

  • Workflow: .github/workflows/e2e.yaml
  • jobs input: mcp-bridge,openclaw-inference-switch

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

E2E Target Recommendation

Required E2E targets: None
Optional E2E targets: None

Workflow run

Full E2E target advisor summary

E2E Target Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E targets

  • None. No live-supported E2E target in e2e.yaml/registry exercises managed vLLM/local-vLLM model defaults or the generic-Linux vLLM serve command path changed here. The PR only changes src/lib/inference vLLM code and colocated unit tests; dispatching the current typed target fan-out would not prove this surface unless a vLLM live target/job is added and wired.

Optional E2E targets

  • None.

Relevant changed files

  • None.

ericksoa added 2 commits July 6, 2026 19:49
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@ericksoa ericksoa added bug-fix PR fixes a bug or regression area: inference Inference routing, serving, model selection, or outputs provider: vllm vLLM local or hosted provider behavior platform: linux Affects non-Ubuntu Linux environments labels Jul 7, 2026
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

E2E Target Results — ✅ All requested jobs passed

Run: 28838185730
Workflow ref: fix/vllm-linux-nemotron-tool-call-parser-6314
Requested targets: (default — all supported)
Requested jobs: inference-routing
Summary: 1 passed, 0 failed, 0 cancelled, 0 skipped

Job Result
inference-routing ✅ success

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

E2E Target Results — ✅ All requested jobs passed

Run: 28838310801
Workflow ref: fix/vllm-linux-nemotron-tool-call-parser-6314
Requested targets: (default — all supported)
Requested jobs: mcp-bridge,openclaw-inference-switch
Summary: 2 passed, 0 failed, 0 cancelled, 0 skipped

Job Result
mcp-bridge ✅ success
openclaw-inference-switch ✅ success

@ericksoa ericksoa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at exact head 4f847e8f76373ab3d62386ea4902a0263a70f353 after current-main conflict audit and the nine-category sensitive-path review. I explicitly accept the unavailable generic-Linux discrete-GPU vLLM proof for this scoped registry fix: deterministic command/profile tests passed (47/47), the full inference suite passed (502 with 1 pre-existing skip), build/typecheck/check:diff and all 49 CI checks passed, inference-routing run 28838185730 passed, and advisor-required mcp-bridge plus both openclaw-inference-switch variants passed in run 28838310801. The change adds fixed argv tokens only and does not alter credentials, authentication, policy, or privilege boundaries.

@cv
cv merged commit 6e2946c into main Jul 7, 2026
202 of 203 checks passed
@cv
cv deleted the fix/vllm-linux-nemotron-tool-call-parser-6314 branch July 7, 2026 07:44
@wscurran wscurran added area: local-models Local model providers, downloads, launch, or connectivity area: providers Inference provider integrations and provider behavior and removed platform: linux Affects non-Ubuntu Linux environments labels Jul 7, 2026
apurvvkumaria added a commit that referenced this pull request Jul 8, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Prepares the user-facing documentation for NemoClaw v0.0.76 and closes
the release-prep documentation gate. It adds the release highlights,
documents the arm64 Local NIM warning and expanded image cleanup
behavior, and fixes agent-specific command headings in generated guides.

## Changes

- Add the v0.0.76 release-notes section and move the shared-gateway
route containment entry out of the v0.0.74 history where it was
incorrectly placed.
- Document the advisory Linux arm64 Local NIM manifest warning in the
canonical platform matrix and local-inference guidance.
- Document that `gc` scans both gateway-built and locally prebuilt
sandbox image repositories.
- Keep OpenClaw and Hermes session headings out of the generated Deep
Agents command guide.
- Add a focused variant regression test for the agent-specific session
headings.

### Source summary

| Merged sources | Documentation coverage |
| --- | --- |
| [#6414](#6414),
[#6418](#6418),
[#6416](#6416),
[#6344](#6344) | v0.0.76 release
notes and the Deep Agents quickstart/inference routes |
| [#6340](#6340) | v0.0.76
release notes and existing Deep Agents observability guidance |
| [#6338](#6338),
[#6378](#6378),
[#6297](#6297) | v0.0.76 release
notes and existing inference/troubleshooting guidance |
| [#6362](#6362) | v0.0.76
release notes and existing lifecycle, command, and credential guidance |
| [#6330](#6330),
[#6307](#6307),
[#6008](#6008) | v0.0.76 release
notes and existing security, troubleshooting, and command guidance |
| [#6382](#6382) | v0.0.76
release notes and existing MCP/command guidance |
| [#6326](#6326),
[#5868](#5868),
[#5539](#5539) | v0.0.76 release
notes, platform matrix, inference options, and local-inference guidance
|
| [#6396](#6396),
[#6390](#6390),
[#6007](#6007) | v0.0.76 release
notes and existing messaging guidance |
| [#5388](#5388),
[#6249](#6249),
[#6303](#6303),
[#6306](#6306) | v0.0.76 release
notes and command/lifecycle guidance |

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run --project integration
test/generate-platform-docs.test.ts test/agent-variant-docs.test.ts
test/sync-agent-variant-docs.test.ts` (3 files, 29 tests passed)
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result:
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
completed with 0 errors and 2 pre-existing Fern warnings
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Added v0.0.76 release notes content, and removed an older conflicting
bullet from the surrounding release history.
* Expanded Local NVIDIA NIM guidance across inference/provider docs,
including an advisory for Linux arm64 DGX Spark/DGX Station hosts when a
matching `linux/arm64` image manifest is unavailable.
* Updated the command reference for correct session-section rendering
and clarified `gc` image cleanup sources.
* **Tests**
* Added coverage ensuring Deep Agents omits sessions headings while
Hermes includes them.
* **CI**
* Refreshed Local NVIDIA NIM provider notes used in the platform matrix.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
…ult (NVIDIA#6326)

<!-- markdownlint-disable MD041 -->
## Summary

The generic-Linux managed-vLLM default,
`nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8`, was launched without
`--enable-auto-tool-choice` or a tool-call parser. Plain completions
succeeded, but agent requests using `tool_choice: "auto"` failed HTTP
400. This PR pins `--tool-call-parser qwen3_coder`, matching NVIDIA's
model-card launch recipe, and adds regression coverage at both the
model-registry and generic-Linux profile layers.

## Related Issue

Closes NVIDIA#6314.

## Reproduction and Analysis

The reporter reproduced the failure on Ubuntu 24.04 x86_64 with an RTX
5070 (12 GB) using NemoClaw v0.0.74:

```bash
NEMOCLAW_EXPERIMENTAL=1 NEMOCLAW_PROVIDER=install-vllm \
  nemoclaw onboard --agent <agent> --name <name> --non-interactive --fresh --yes
```

Before this fix, `buildVllmServeCommand` produced a command ending with:

```text
... --gpu-memory-utilization 0.7 --load-format fastsafetensors
```

The reporter observed:

```text
PLAIN:    HTTP_CODE=200
TOOLCALL: HTTP_CODE=400
"auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set
```

`GENERIC_LINUX_PROFILE.defaultModel` resolves to the
`nemotron-3-nano-4b` registry entry. That entry was the outlier: the DGX
Spark (`qwen3.6-35b-a3b-nvfp4` / `qwen3_xml`) and DGX Station
(`deepseek-v4-flash` / `deepseek_v4`) defaults already pinned their own
tool-call parsers.

After this fix, the command ends with:

```text
... --gpu-memory-utilization 0.7 --load-format fastsafetensors \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder
```

The parser choice comes from the NVIDIA Nemotron-3-Nano-4B-FP8
model-card vLLM example and uses a parser already present in this
registry.

## Changes

- Add `--enable-auto-tool-choice --tool-call-parser qwen3_coder` to the
Nemotron-3-Nano-4B FP8 registry entry.
- Cover the generated serve command and the generic-Linux profile
default with regression tests.
- Require each tool-call switch to occur exactly once in the generated
command.
- Merge current `main` into the contributor branch without rewriting
contributor history.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: this corrects the baked serve
arguments for an existing managed-vLLM model and requires no new user
action; current docs already describe the generic-Linux default,
managed-vLLM limitations, and the need for model-appropriate tool-call
flags.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: maintainer review by
@ericksoa on the final diff (2026-07-06) covered all nine
security-review categories. The change adds fixed model-specific argv
tokens, does not introduce user-controlled interpolation or alter
credential/auth/policy boundaries, retains negative-path parser
coverage, and passed gitleaks; no security findings.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable —
`npm run check:diff` passed at
`4f847e8f76373ab3d62386ea4902a0263a70f353`.
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run
src/lib/inference/vllm-models.test.ts src/lib/inference/vllm.test.ts`
(47/47 passed).
- [x] Applicable broad gate passed — `npx vitest run src/lib/inference/`
(502 passed, 1 pre-existing skip), `npm run typecheck:cli`, and `npm run
build:cli` passed.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

## Remaining Runtime Evidence

Live end-to-end proof on an x86_64 generic-Linux NVIDIA GPU host is not
included. The available fleet does not have that architecture/GPU
combination; the DGX Spark host exercises a different profile that was
already unaffected. The unit tests prove the exact command/profile
composition, and the required `inference-routing` E2E is being run at
the final head, but that E2E does not launch managed vLLM or validate
this parser against the model. Before merge, a maintainer must either
explicitly accept this hardware-evidence limitation or obtain a
generic-Linux GPU run showing the fixed command and a successful
`tool_choice: "auto"` request.

## AI Disclosure

- [x] AI-assisted — contributor tool: Claude Code; maintainer salvage
tool: Codex

---
Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Updated vLLM settings for the Nemotron-3-Nano-4B FP8 model to enable
automatic tool choice with the correct tool-call parser configuration.
* Fixed Linux/NVIDIA vLLM profile detection to include the expected
tool-call parser settings.
* **Tests**
* Added regression coverage to ensure the tool-call related flags are
included correctly going forward.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Co-authored-by: Aaron Erickson <aerickson@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Prepares the user-facing documentation for NemoClaw v0.0.76 and closes
the release-prep documentation gate. It adds the release highlights,
documents the arm64 Local NIM warning and expanded image cleanup
behavior, and fixes agent-specific command headings in generated guides.

## Changes

- Add the v0.0.76 release-notes section and move the shared-gateway
route containment entry out of the v0.0.74 history where it was
incorrectly placed.
- Document the advisory Linux arm64 Local NIM manifest warning in the
canonical platform matrix and local-inference guidance.
- Document that `gc` scans both gateway-built and locally prebuilt
sandbox image repositories.
- Keep OpenClaw and Hermes session headings out of the generated Deep
Agents command guide.
- Add a focused variant regression test for the agent-specific session
headings.

### Source summary

| Merged sources | Documentation coverage |
| --- | --- |
| [NVIDIA#6414](NVIDIA#6414),
[NVIDIA#6418](NVIDIA#6418),
[NVIDIA#6416](NVIDIA#6416),
[NVIDIA#6344](NVIDIA#6344) | v0.0.76 release
notes and the Deep Agents quickstart/inference routes |
| [NVIDIA#6340](NVIDIA#6340) | v0.0.76
release notes and existing Deep Agents observability guidance |
| [NVIDIA#6338](NVIDIA#6338),
[NVIDIA#6378](NVIDIA#6378),
[NVIDIA#6297](NVIDIA#6297) | v0.0.76 release
notes and existing inference/troubleshooting guidance |
| [NVIDIA#6362](NVIDIA#6362) | v0.0.76
release notes and existing lifecycle, command, and credential guidance |
| [NVIDIA#6330](NVIDIA#6330),
[NVIDIA#6307](NVIDIA#6307),
[NVIDIA#6008](NVIDIA#6008) | v0.0.76 release
notes and existing security, troubleshooting, and command guidance |
| [NVIDIA#6382](NVIDIA#6382) | v0.0.76
release notes and existing MCP/command guidance |
| [NVIDIA#6326](NVIDIA#6326),
[NVIDIA#5868](NVIDIA#5868),
[NVIDIA#5539](NVIDIA#5539) | v0.0.76 release
notes, platform matrix, inference options, and local-inference guidance
|
| [NVIDIA#6396](NVIDIA#6396),
[NVIDIA#6390](NVIDIA#6390),
[NVIDIA#6007](NVIDIA#6007) | v0.0.76 release
notes and existing messaging guidance |
| [NVIDIA#5388](NVIDIA#5388),
[NVIDIA#6249](NVIDIA#6249),
[NVIDIA#6303](NVIDIA#6303),
[NVIDIA#6306](NVIDIA#6306) | v0.0.76 release
notes and command/lifecycle guidance |

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run --project integration
test/generate-platform-docs.test.ts test/agent-variant-docs.test.ts
test/sync-agent-variant-docs.test.ts` (3 files, 29 tests passed)
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result:
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
completed with 0 errors and 2 pre-existing Fern warnings
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Added v0.0.76 release notes content, and removed an older conflicting
bullet from the surrounding release history.
* Expanded Local NVIDIA NIM guidance across inference/provider docs,
including an advisory for Linux arm64 DGX Spark/DGX Station hosts when a
matching `linux/arm64` image manifest is unavailable.
* Updated the command reference for correct session-section rendering
and clarified `gc` image cleanup sources.
* **Tests**
* Added coverage ensuring Deep Agents omits sessions headings while
Hermes includes them.
* **CI**
* Refreshed Local NVIDIA NIM provider notes used in the platform matrix.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: inference Inference routing, serving, model selection, or outputs area: local-models Local model providers, downloads, launch, or connectivity area: providers Inference provider integrations and provider behavior bug-fix PR fixes a bug or regression provider: vllm vLLM local or hosted provider behavior

Projects

None yet

4 participants