Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
f709b28
feat(inference): add fixed local serving profiles
ericksoa Aug 5, 2026
d931d52
feat(inference): add serving profile discovery (#8384)
prekshivyas Aug 5, 2026
888d557
docs(inference): name profile conflict variable (#8384)
prekshivyas Aug 5, 2026
ff9f532
refactor(onboard): keep local profile wiring modular
ericksoa Aug 5, 2026
d7ab743
docs(cli): include profile on setup aliases (#8384)
prekshivyas Aug 5, 2026
d9f4437
test(inference): keep local profile fixtures branch-free
ericksoa Aug 5, 2026
39a8e6f
fix(inference): harden profile selection boundaries (#8384)
prekshivyas Aug 5, 2026
12468a7
fix(onboard): restore scoped manifest environment (#8384)
prekshivyas Aug 5, 2026
293c195
fix(inference): fail closed on unsafe local state
ericksoa Aug 5, 2026
b632a2d
chore: merge main into local serving profiles
ericksoa Aug 5, 2026
ce244fd
test(onboard): use canonical command environment (#8384)
prekshivyas Aug 5, 2026
32f732f
test(uninstall): reject unexpected Docker mocks
ericksoa Aug 5, 2026
f3d2b94
chore: merge main into local serving profiles
ericksoa Aug 5, 2026
bf63e70
fix(inference): key runtime auth fingerprints
ericksoa Aug 5, 2026
2cff408
fix(onboard): restore profile env on manifest failure
prekshivyas Aug 5, 2026
0769f31
docs(inference): clarify profile preset conflicts
prekshivyas Aug 5, 2026
1c3bc30
fix(inference): complete local profile lifecycle validation
prekshivyas Aug 5, 2026
95d0dfb
chore: merge main into local serving profiles
prekshivyas Aug 5, 2026
49d6254
test(inference): satisfy strict profile fixture types
prekshivyas Aug 5, 2026
491c52f
chore: integrate fixed local serving profiles
prekshivyas Aug 5, 2026
61d65c8
test(inference): keep readiness fixtures linear
prekshivyas Aug 6, 2026
7c25b61
merge: resolve conflicts with main
github-actions[bot] Aug 6, 2026
764da09
feat(inference): persist serving profile provenance
prekshivyas Aug 6, 2026
725ea6c
Merge branch 'feat/fixed-local-serving-profiles' of https://github.co…
prekshivyas Aug 6, 2026
e5fff96
fix(inference): harden profile selection diagnostics
prekshivyas Aug 6, 2026
6c5cc6d
chore(onboard): keep entrypoint growth neutral
prekshivyas Aug 6, 2026
d01b970
test(inference): mock verified vllm startup boundaries
prekshivyas Aug 6, 2026
b235cfc
test(inference): verify storage fixture container identity
prekshivyas Aug 6, 2026
29eee55
ci: retry required e2e after hosted inference timeout
prekshivyas Aug 6, 2026
583edfa
merge: resolve conflicts with main
prekshivyas Aug 6, 2026
5d8f6e3
merge: resolve conflicts with main
github-actions[bot] Aug 6, 2026
33eb0c0
merge: preserve concurrent conflict resolution
prekshivyas Aug 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ci/source-architecture-budget.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"src/lib/adapters/openshell/timeouts.ts": 37,
"src/lib/agent/defs.ts": 32,
"src/lib/cli/branding.ts": 87,
"src/lib/cli/nemoclaw-oclif-command.ts": 104,
"src/lib/cli/nemoclaw-oclif-command.ts": 105,
"src/lib/cli/terminal-style.ts": 45,
"src/lib/core/json-types.ts": 37,
"src/lib/core/ports.ts": 88,
Expand Down
27 changes: 27 additions & 0 deletions docs/inference/set-up-vllm.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,33 @@ Without both `--enable-auto-tool-choice` and a matching parser, tool calls can a
NemoClaw displays the **Install vLLM** or **Start vLLM** entry by default on DGX Spark and DGX Station.
Generic Linux NVIDIA GPU hosts require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm` before the managed entry appears.

### Discover Serving Profiles

List the serving profiles installed with NemoClaw before you select a managed inference configuration.

```bash
$$nemoclaw profiles list
```

The command evaluates the catalog against the current host without downloading a model or changing host, gateway, inference, or sandbox resources.
Each entry includes a stable ID, display name, backend, model, topology, selection mode, support state, estimated downloads, and an incompatibility reason when applicable.
Add `--json` when a script needs the same fields as structured output.

### Select a Serving Profile

Pass a compatible profile's stable ID to onboarding.

```bash
$$nemoclaw onboard --profile <profile-id>
```

The same profile selector works with interactive and non-interactive onboarding.
NemoClaw rejects unknown, ambiguous, disabled, incompatible, or conflicting selections before image or model downloads begin.
Do not combine `--profile` with `NEMOCLAW_PROVIDER`, `NEMOCLAW_MODEL`, `NEMOCLAW_VLLM_MODEL`, `NEMOCLAW_SERVING_PRESET`, `NEMOCLAW_MANAGED_CLUSTER_PEERS`, or `NEMOCLAW_VLLM_EXTRA_ARGS_JSON` overrides.

The profile selector is model-independent, so a catalog can add profiles without adding model-specific CLI flags.
Omitting `--profile` preserves the existing provider and model defaults.

In an interactive run, the managed path lists the supported model registry for your host profile.
Press **Enter** to use the profile default or select another validated model with its registered `vllm serve` arguments.

Expand Down
37 changes: 36 additions & 1 deletion docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -205,14 +205,33 @@ hermes Self-improving AI agent with learning loop (Nous Rese
langchain-deepagents-code Terminal coding agent built on the Deep Agents SDK
```

### `$$nemoclaw profiles list`

List the serving profiles installed with NemoClaw and evaluate them against the current host.
The command reports each profile's stable ID, display name, inference backend, model, topology, selection mode, support state, estimated downloads, and incompatibility reason.
It reads the serving catalog and host readiness state without downloading a model or changing host, gateway, inference, or sandbox resources.

```bash
$$nemoclaw profiles list
```

Use `--json` for machine-readable output with the same profile fields.

```bash
$$nemoclaw profiles list --json
```

Use the stable `id` value with `$$nemoclaw onboard --profile <name>`.
Display names are accepted when they identify exactly one profile, but stable IDs are suitable for scripts and automation.

### `$$nemoclaw onboard`

Run the interactive setup wizard (recommended for new installs).
The wizard creates an OpenShell gateway, registers inference providers, builds the sandbox image, and creates the sandbox.
Use this command for new installs and for recreating a sandbox after changes to policy or configuration.

```bash
$$nemoclaw onboard [--non-interactive] [--resume | --fresh] [--recreate-sandbox] [--gpu | --no-gpu] [--from <Dockerfile>] [--name <sandbox>] [--sandbox-gpu | --no-sandbox-gpu] [--sandbox-gpu-device <device>] [--agent <name>] [--agents <agents.yaml>] [--tool-disclosure <progressive|direct>] [--observability | --no-observability] [--control-ui-port <N>] [--events=jsonl] [--yes | -y] [--no-ollama-autostart] [--yes-i-accept-third-party-software]
$$nemoclaw onboard [--profile <name>] [--non-interactive] [--resume | --fresh] [--recreate-sandbox] [--gpu | --no-gpu] [--from <Dockerfile>] [--name <sandbox>] [--sandbox-gpu | --no-sandbox-gpu] [--sandbox-gpu-device <device>] [--agent <name>] [--agents <agents.yaml>] [--tool-disclosure <progressive|direct>] [--observability | --no-observability] [--control-ui-port <N>] [--events=jsonl] [--yes | -y] [--no-ollama-autostart] [--yes-i-accept-third-party-software]
```

<AgentOnly variant="hermes">
Expand All @@ -239,6 +258,22 @@ nemoclaw onboard --agent langchain-deepagents-code [options]
`--agent` accepts the canonical manifest names from `$$nemoclaw agents list` plus common aliases.
For example, `nemohermes` resolves to `hermes`, while `dcode`, `deepagents`, `deepagents-code`, and `langchain` resolve to `langchain-deepagents-code`.

#### `--profile <name>`

Select one serving profile from `$$nemoclaw profiles list` for interactive or non-interactive onboarding.
The flag is generic and does not add a model-specific command or flag.
NemoClaw maps a unique display name to its stable catalog ID and passes that ID to the managed inference path.

```bash
$$nemoclaw onboard --profile <profile-id>
```

NemoClaw rejects an unknown, ambiguous, disabled, or incompatible profile before image or model downloads begin.
It also rejects `--profile` when you combine it with `NEMOCLAW_PROVIDER`, `NEMOCLAW_MODEL`, `NEMOCLAW_VLLM_MODEL`, `NEMOCLAW_SERVING_PRESET`, `NEMOCLAW_MANAGED_CLUSTER_PEERS`, or `NEMOCLAW_VLLM_EXTRA_ARGS_JSON` overrides.
Run `$$nemoclaw profiles list` to inspect an incompatibility reason before onboarding.

If you omit `--profile`, onboarding uses the same provider and model defaults as an installation without this feature.

#### `--events=jsonl`

Emit a read-only stream of canonical onboarding FSM events as JSON Lines on stdout.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ kind: ServingPreset
metadata:
id: llama-cpp.dgx-spark-gb10.single.nemotron-3-nano-30b-a3b
displayName: NVIDIA Nemotron 3 Nano 30B-A3B on one DGX Spark
supportState: experimental

spec:
selection: explicit-only
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ kind: ServingPreset
metadata:
id: vllm.dgx-spark-gb10.dual.deepseek-v4-flash-0731
displayName: DeepSeek V4 Flash 0731 on two DGX Sparks
supportState: experimental

spec:
selection: automatic
Expand Down
7 changes: 7 additions & 0 deletions managed-inference/schemas/preset.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,13 @@
"type": "string",
"minLength": 1,
"maxLength": 128
},
"supportState": {
"enum": [
"supported",
"experimental",
"disabled"
]
}
},
"additionalProperties": false
Expand Down
37 changes: 37 additions & 0 deletions src/commands/profiles/list.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import { NemoClawCommand } from "../../lib/cli/nemoclaw-oclif-command";
import {
listServingProfiles,
renderServingProfiles,
} from "../../lib/inference/serving/profile-list";

export default class ProfilesListCommand extends NemoClawCommand {
static id = "profiles:list";
static strict = true;
static enableJsonFlag = true;
static summary = "List installed serving profiles and evaluate host compatibility";
static description =
"Inspect the installed serving catalog and current host without downloading models or changing host or runtime state.";
static usage = ["profiles list [--json]"];
static examples = ["<%= config.bin %> profiles list", "<%= config.bin %> profiles list --json"];
static flags = {};
static publicDisplay = [
{
usage: "nemoclaw profiles list",
description: "List installed serving profiles and evaluate host compatibility",
flags: "[--json]",
group: "Getting Started",
scope: "global",
order: 1.6,
},
] as const;
Comment thread
prekshivyas marked this conversation as resolved.
Resolved

public async run(): Promise<unknown> {
await this.parse(ProfilesListCommand);
const entries = listServingProfiles();
if (this.jsonEnabled()) return entries;
this.log(renderServingProfiles(entries));
}
}
56 changes: 56 additions & 0 deletions src/lib/inference/serving/profile-list.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import { describe, expect, it } from "vitest";

import { loadServingCatalog } from "./catalog-loader";
import { listServingProfiles, renderServingProfiles } from "./profile-list";

describe("serving profile discovery", () => {
it("lists every compiled preset with stable selection metadata (#8384)", () => {
const catalog = loadServingCatalog();
const entries = listServingProfiles(catalog);
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

expect(entries.map(({ id }) => id)).toEqual(
[...catalog.presets].map(({ metadata }) => metadata.id).sort(),
);
for (const entry of entries) {
expect(entry).toMatchObject({
id: expect.any(String),
displayName: expect.any(String),
backend: expect.any(String),
model: expect.any(String),
topology: expect.any(String),
selectionMode: expect.stringMatching(/^(automatic|explicit-only|disabled)$/u),
supportState: expect.stringMatching(/^(supported|experimental|disabled)$/u),
compatible: expect.any(Boolean),
});
expect(
entry.compatible ? entry.incompatibilityReason : typeof entry.incompatibilityReason,
).toBe(entry.compatible ? null : "string");
}
});

it("renders IDs, selection state, support state, and compatibility (#8384)", () => {
const output = renderServingProfiles([
{
id: "vllm.spark.example",
displayName: "Example profile",
backend: "vllm",
model: "example/model",
topology: "single-host",
selectionMode: "explicit-only",
supportState: "experimental",
estimatedImageDownloadBytes: 2 * 1024 ** 3,
estimatedModelDownloadBytes: 3 * 1024 ** 3,
compatible: false,
incompatibilityReason: "Host requirement is not met.",
},
]);

expect(output).toContain("vllm.spark.example Example profile");
expect(output).toContain("selection=explicit-only support=experimental");
expect(output).toContain("image=2.0 GiB model-download=3.0 GiB");
expect(output).toContain("incompatible: Host requirement is not met.");
});
});
161 changes: 161 additions & 0 deletions src/lib/inference/serving/profile-list.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import os from "node:os";

import { getBuildIdentity } from "../../core/version.js";
import { createHostReadinessReport } from "../../readiness/host.js";
import { loadServingCatalog, managedInferenceCatalogFromServingCatalog } from "./catalog-loader.js";
import { resolveManagedInferenceServing } from "./resolver.js";
import type {
CompiledServingCatalog,
ServingPreset,
ServingRecipe,
ServingSelectionPolicy,
ServingSupportState,
} from "./types.js";

export interface ServingProfileListEntry {
readonly id: string;
readonly displayName: string;
readonly backend: string;
readonly model: string;
readonly topology: string;
readonly selectionMode: ServingSelectionPolicy;
readonly supportState: ServingSupportState;
readonly estimatedImageDownloadBytes: number | null;
readonly estimatedModelDownloadBytes: number | null;
readonly compatible: boolean;
readonly incompatibilityReason: string | null;
}

function recipeForPreset(
catalog: CompiledServingCatalog,
preset: ServingPreset,
): ServingRecipe | undefined {
return catalog.recipes.find(({ metadata }) => metadata.id === preset.spec.plan.recipeRef);
}

function modelDownloadBytes(recipe: ServingRecipe): number | null {
if (
"downloadSizeBytes" in recipe.spec.model &&
typeof recipe.spec.model.downloadSizeBytes === "number"
) {
return recipe.spec.model.downloadSizeBytes;
}
const sizes = recipe.spec.model.files?.map((file) =>
"sizeBytes" in file && typeof file.sizeBytes === "number" ? file.sizeBytes : null,
);
return sizes?.length && sizes.every((size): size is number => size !== null)
? sizes.reduce((total, size) => total + size, 0)
: null;
}

function topologyLabel(recipe: ServingRecipe): string {
const executionHosts = recipe.spec.execution.nodeCount;
const runtimeHosts =
recipe.spec.runtime && "hosts" in recipe.spec.runtime ? recipe.spec.runtime.hosts : undefined;
const hosts = executionHosts ?? runtimeHosts ?? 1;
return hosts === 1 ? "single-host" : `${String(hosts)}-host`;
}

function supportState(preset: ServingPreset): ServingSupportState {
if (preset.spec.selection === "disabled") return "disabled";
return preset.metadata.supportState ?? "experimental";
}

function compatibility(
catalog: CompiledServingCatalog,
preset: ServingPreset,
recipe: ServingRecipe,
): { compatible: boolean; incompatibilityReason: string | null } {
if (preset.spec.selection === "disabled" || supportState(preset) === "disabled") {
return { compatible: false, incompatibilityReason: "Profile is disabled." };
}
if (recipe.spec.backend !== "vllm") {
return {
compatible: false,
incompatibilityReason: `Backend ${recipe.spec.backend} is not available through Express onboarding yet.`,
};
}
const resolution = resolveManagedInferenceServing(
{
readinessReports: [
{
nodeId: os.hostname(),
report: createHostReadinessReport(getBuildIdentity()),
},
],
topologyQualifications: [],
intent: { preset: preset.metadata.id },
},
managedInferenceCatalogFromServingCatalog(catalog),
);
return resolution.outcome === "selected"
? { compatible: true, incompatibilityReason: null }
: { compatible: false, incompatibilityReason: resolution.message };
}

export function listServingProfiles(
catalog: CompiledServingCatalog = loadServingCatalog(),
): ServingProfileListEntry[] {
return [...catalog.presets]
.sort((left, right) => left.metadata.id.localeCompare(right.metadata.id))
.map((preset) => {
const recipe = recipeForPreset(catalog, preset);
if (!recipe) {
return {
id: preset.metadata.id,
displayName: preset.metadata.displayName ?? preset.metadata.id,
backend: preset.spec.plan.backend,
model: "unknown",
topology: "unknown",
selectionMode: preset.spec.selection,
supportState: supportState(preset),
estimatedImageDownloadBytes: null,
estimatedModelDownloadBytes: null,
compatible: false,
incompatibilityReason: `Recipe ${preset.spec.plan.recipeRef} is unavailable.`,
};
}
const runtime = recipe.spec.runtime;
const imageDownloadBytes =
runtime && "imageDownloadSizeBytes" in runtime
? (runtime.imageDownloadSizeBytes ?? null)
: null;
return {
id: preset.metadata.id,
displayName: preset.metadata.displayName ?? preset.metadata.id,
backend: recipe.spec.backend,
model: recipe.spec.model.id,
topology: topologyLabel(recipe),
selectionMode: preset.spec.selection,
supportState: supportState(preset),
estimatedImageDownloadBytes: imageDownloadBytes,
estimatedModelDownloadBytes: modelDownloadBytes(recipe),
...compatibility(catalog, preset, recipe),
};
});
}

function formatBytes(value: number | null): string {
if (value === null) return "unknown";
return `${(value / 1024 ** 3).toFixed(1)} GiB`;
}

export function renderServingProfiles(entries: readonly ServingProfileListEntry[]): string {
if (entries.length === 0) return "No serving profiles are installed.";
return entries
.map((entry) => {
const availability = entry.compatible
? "compatible"
: `incompatible: ${entry.incompatibilityReason ?? "unknown reason"}`;
return [
`${entry.id} ${entry.displayName}`,
` backend=${entry.backend} model=${entry.model} topology=${entry.topology}`,
` selection=${entry.selectionMode} support=${entry.supportState} image=${formatBytes(entry.estimatedImageDownloadBytes)} model-download=${formatBytes(entry.estimatedModelDownloadBytes)}`,
` ${availability}`,
].join("\n");
})
.join("\n\n");
}
Loading
Loading