Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/inference/local-compatible-inference-setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,8 @@ On supported Linux hosts with NVIDIA GPUs, the onboard wizard can also install o

### Use an Existing vLLM Server

For an already-running vLLM server, run `$$nemoclaw onboard` and select **Local vLLM [experimental]** from the provider list.
For an already-running vLLM server, run `$$nemoclaw onboard` and select **Local vLLM** from the provider list.
On generic hosts, you see an experimental label on the entry; on DGX Spark or DGX Station, you do not.
If vLLM is already running, NemoClaw detects the running model and validates the endpoint.
When vLLM exposes runtime metadata such as `max_model_len`, NemoClaw uses that value for the configured context window unless you set `NEMOCLAW_CONTEXT_WINDOW` yourself.

Expand Down
3 changes: 2 additions & 1 deletion docs/inference/tool-calling-reliability.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,8 @@ These routes therefore keep direct structured tool calling until search, describ

## Recommended Fix

For persistent NemoClaw use, start vLLM with auto tool choice and the parser that matches your model family, then rerun onboarding and select **Local vLLM [experimental]** or **Other OpenAI-compatible endpoint**.
For persistent NemoClaw use, start vLLM with auto tool choice and the parser that matches your model family, then rerun onboarding and select **Local vLLM** or **Other OpenAI-compatible endpoint**.
On generic hosts, you see an experimental label on the Local vLLM entry; on DGX Spark or DGX Station, you do not.

### Configure vLLM

Expand Down
24 changes: 23 additions & 1 deletion src/lib/onboard/vllm-menu.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,12 @@ describe("buildVllmMenuEntries", () => {
assert.deepEqual(entries, []);
});

it("returns the running entry when vLLM is reachable on localhost", () => {
it("marks the running entry experimental on generic hosts", () => {
const entries = buildVllmMenuEntries({
vllmRunning: true,
vllmProfile: null,
experimental: false,
platform: "linux",
hasVllmImage: false,
log: () => {},
env: {},
Expand All @@ -35,6 +36,27 @@ describe("buildVllmMenuEntries", () => {
assert.match(entries[0].label, /running/);
});

for (const [platform, hostLabel] of [
["spark", "Spark"],
["station", "Station"],
] as const) {
it(`does not mark the running entry experimental on DGX ${hostLabel}`, () => {
const entries = buildVllmMenuEntries({
vllmRunning: true,
vllmProfile: null,
experimental: false,
platform,
hasVllmImage: false,
log: () => {},
env: {},
});
assert.equal(entries.length, 1);
assert.equal(entries[0].key, "vllm");
assert.doesNotMatch(entries[0].label, /experimental/);
assert.match(entries[0].label, /running/);
});
}

it("returns the install entry when a profile matches and EXPERIMENTAL is set", () => {
const entries = buildVllmMenuEntries({
vllmRunning: false,
Expand Down
4 changes: 3 additions & 1 deletion src/lib/onboard/vllm-menu.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,12 @@ export function buildVllmMenuEntries(opts: BuildVllmMenuOptions): VllmMenuEntry[
` Note: NEMOCLAW_PROVIDER=install-vllm requested, but vLLM is already running on localhost:${VLLM_PORT} — selecting the running instance.`,
);
}
const experimentalLabel =
opts.platform && MANAGED_VLLM_DEFAULT_PLATFORMS.has(opts.platform) ? "" : " [experimental]";
return [
{
key: "vllm",
label: `Local vLLM [experimental] (localhost:${VLLM_PORT}) — running (suggested)`,
label: `Local vLLM${experimentalLabel} (localhost:${VLLM_PORT}) — running (suggested)`,
},
];
}
Expand Down
Loading