Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/inference/local-compatible-inference-setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@ On supported Linux hosts with NVIDIA GPUs, the onboard wizard can also install o

### Use an Existing vLLM Server

For an already-running vLLM server, run `$$nemoclaw onboard` and select **Local vLLM [experimental]** from the provider list.
For an already-running vLLM server, run `$$nemoclaw onboard` and select **Local vLLM** from the provider list. The entry retains an experimental label on generic hosts, but not on DGX Spark or DGX Station.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
If vLLM is already running, NemoClaw detects the running model and validates the endpoint.
When vLLM exposes runtime metadata such as `max_model_len`, NemoClaw uses that value for the configured context window unless you set `NEMOCLAW_CONTEXT_WINDOW` yourself.

Expand Down
2 changes: 1 addition & 1 deletion docs/inference/tool-calling-reliability.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ These routes therefore keep direct structured tool calling until search, describ

## Recommended Fix

For persistent NemoClaw use, start vLLM with auto tool choice and the parser that matches your model family, then rerun onboarding and select **Local vLLM [experimental]** or **Other OpenAI-compatible endpoint**.
For persistent NemoClaw use, start vLLM with auto tool choice and the parser that matches your model family, then rerun onboarding and select **Local vLLM** or **Other OpenAI-compatible endpoint**. The Local vLLM entry retains an experimental label on generic hosts, but not on DGX Spark or DGX Station.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

### Configure vLLM

Expand Down
23 changes: 22 additions & 1 deletion src/lib/onboard/vllm-menu.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ describe("buildVllmMenuEntries", () => {
assert.deepEqual(entries, []);
});

it("returns the running entry when vLLM is reachable on localhost", () => {
it("marks the running entry experimental on generic hosts", () => {
const entries = buildVllmMenuEntries({
vllmRunning: true,
vllmProfile: null,
Expand All @@ -35,6 +35,27 @@ describe("buildVllmMenuEntries", () => {
assert.match(entries[0].label, /running/);
});

for (const [platform, hostLabel] of [
["spark", "Spark"],
["station", "Station"],
] as const) {
it(`does not mark the running entry experimental on DGX ${hostLabel}`, () => {
const entries = buildVllmMenuEntries({
vllmRunning: true,
vllmProfile: null,
experimental: false,
platform,
hasVllmImage: false,
log: () => {},
env: {},
});
assert.equal(entries.length, 1);
assert.equal(entries[0].key, "vllm");
assert.doesNotMatch(entries[0].label, /experimental/);
assert.match(entries[0].label, /running/);
});
}

it("returns the install entry when a profile matches and EXPERIMENTAL is set", () => {
const entries = buildVllmMenuEntries({
vllmRunning: false,
Expand Down
4 changes: 3 additions & 1 deletion src/lib/onboard/vllm-menu.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,12 @@ export function buildVllmMenuEntries(opts: BuildVllmMenuOptions): VllmMenuEntry[
` Note: NEMOCLAW_PROVIDER=install-vllm requested, but vLLM is already running on localhost:${VLLM_PORT} — selecting the running instance.`,
);
}
const experimentalLabel =
opts.platform && MANAGED_VLLM_DEFAULT_PLATFORMS.has(opts.platform) ? "" : " [experimental]";
return [
{
key: "vllm",
label: `Local vLLM [experimental] (localhost:${VLLM_PORT}) — running (suggested)`,
label: `Local vLLM${experimentalLabel} (localhost:${VLLM_PORT}) — running (suggested)`,
},
];
}
Expand Down
Loading