Skip to content
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,15 +138,15 @@ Alternatively, send a single message and print the response:
openclaw agent --agent main --local -m "hello" --session-id test
```

### Model Router (Complexity-Based Routing)
### Model Router (Experimental)

NemoClaw includes an optional model router that automatically picks the most efficient model for each query. Instead of sending every request to a single large model, the router uses a lightweight encoder to predict which model in a pool can handle each query correctly, then routes to the cheapest one that meets an accuracy threshold.

The router uses the [NVIDIA LLM Router v3](https://github.com/NVIDIA-AI-Blueprints/llm-router/tree/v3) prefill routing engine and runs on the host as a LiteLLM proxy. The sandbox reaches it through the OpenShell gateway and continues to call `https://inference.local/v1`; do not probe `localhost:4000` or `host.openshell.internal` directly from inside the sandbox.

#### Enable during onboard

Select **Model Router (complexity-based routing)** during the onboard wizard, or set `NEMOCLAW_PROVIDER=routed` for non-interactive mode:
Select **Model Router (experimental)** during the onboard wizard, or set `NEMOCLAW_PROVIDER=routed` for non-interactive mode:

```bash
NEMOCLAW_PROVIDER=routed nemoclaw onboard --non-interactive
Expand Down
4 changes: 2 additions & 2 deletions docs/get-started/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ The inference provider prompt presents a numbered list.
5) Other Anthropic-compatible endpoint
6) Google Gemini
7) Local Ollama (localhost:11434)
8) Model Router (complexity-based routing)
8) Model Router (experimental)
Choose [1]:
```

Expand Down Expand Up @@ -218,7 +218,7 @@ Use `NVIDIA_API_KEY` for the model pool credentials.

Respond to the wizard as follows.

1. At the `Choose [1]:` prompt, type `8` to select **Model Router (complexity-based routing)**.
1. At the `Choose [1]:` prompt, type `8` to select **Model Router (experimental)**.
2. At the `NVIDIA_API_KEY:` prompt, paste your key if it is not already exported.
3. Review the configuration summary and continue with the sandbox build.

Expand Down
50 changes: 1 addition & 49 deletions scripts/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -533,7 +533,7 @@ usage() {
printf " NEMOCLAW_RECREATE_SANDBOX=1 Recreate an existing sandbox\n"
printf " NEMOCLAW_INSTALL_TAG Git ref to install (default: latest release)\n"
printf " NEMOCLAW_PROVIDER build | openai | anthropic | anthropicCompatible\n"
printf " | gemini | ollama | custom | nim-local | vllm\n"
printf " | gemini | ollama | custom | nim-local | vllm | routed\n"
printf " (aliases: cloud -> build, nim -> nim-local)\n"
printf " NEMOCLAW_MODEL Inference model to configure\n"
printf " NEMOCLAW_POLICY_MODE suggested | custom | skip\n"
Expand Down Expand Up @@ -1320,44 +1320,6 @@ is_source_checkout() {
return 1
}

init_nemoclaw_submodules() {
local root="$1"
[[ -f "$root/.gitmodules" ]] || return 0
git -C "$root" rev-parse --git-dir >/dev/null 2>&1 || return 0
git -C "$root" submodule update --init --depth 1 2>/dev/null
}

is_routed_provider_requested() {
local provider="${NEMOCLAW_PROVIDER:-}"
provider="$(printf '%s' "$provider" | tr '[:upper:]' '[:lower:]')"
[[ "$provider" == "routed" ]]
}

install_model_router_if_present() {
local root="$1"
local router_dir="$root/nemoclaw-blueprint/router/llm-router"

if ! command_exists pip3; then
is_routed_provider_requested && error "pip3 is required for routed inference."
return 0
fi
if [[ ! -d "$router_dir" ]]; then
is_routed_provider_requested && error "llm-router is required for routed inference but is missing."
return 0
fi
if [[ ! -f "$router_dir/pyproject.toml" && ! -f "$router_dir/setup.py" ]]; then
is_routed_provider_requested && error "llm-router is required for routed inference but is not initialized."
warn "Skipping model router install — llm-router submodule is not initialized."
return 0
fi
if ! spin "Installing model router" pip3 install --quiet --user "${router_dir}[prefill,proxy]"; then
if is_routed_provider_requested; then
error "pip3 install of llm-router failed"
fi
warn "Skipping model router install — pip3 install failed"
fi
}

install_nemoclaw() {
command_exists git || error "git was not found on PATH."
local repo_root package_json
Expand All @@ -1373,14 +1335,9 @@ install_nemoclaw() {
spin "Preparing OpenClaw package" bash -c "$(declare -f info warn resolve_openclaw_version pre_extract_openclaw); pre_extract_openclaw \"\$1\"" _ "$NEMOCLAW_SOURCE_ROOT" \
|| warn "Pre-extraction failed — npm install may fail if openclaw tarball is broken"
fi
if ! spin "Initializing ${_CLI_DISPLAY} submodules" init_nemoclaw_submodules "$NEMOCLAW_SOURCE_ROOT"; then
is_routed_provider_requested && error "Failed to initialize the llm-router submodule required for routed inference."
warn "Submodule initialization failed — model router support may be unavailable"
fi
spin "Installing ${_CLI_DISPLAY} dependencies" bash -c "cd \"$NEMOCLAW_SOURCE_ROOT\" && npm install --ignore-scripts"
spin "Building ${_CLI_DISPLAY} CLI modules" bash -c "cd \"$NEMOCLAW_SOURCE_ROOT\" && npm run --if-present build:cli"
spin "Building ${_CLI_DISPLAY} plugin" bash -c "cd \"$NEMOCLAW_SOURCE_ROOT\"/nemoclaw && npm install --ignore-scripts && npm run build"
install_model_router_if_present "$NEMOCLAW_SOURCE_ROOT"
spin "Linking ${_CLI_DISPLAY} CLI" bash -c "cd \"$NEMOCLAW_SOURCE_ROOT\" && npm link"
else
if [[ -f "$package_json" ]]; then
Expand All @@ -1399,10 +1356,6 @@ install_nemoclaw() {
mkdir -p "$(dirname "$nemoclaw_src")"
NEMOCLAW_SOURCE_ROOT="$nemoclaw_src"
spin "Cloning ${_CLI_DISPLAY} source" clone_nemoclaw_ref "$release_ref" "$nemoclaw_src"
if ! spin "Initializing ${_CLI_DISPLAY} submodules" init_nemoclaw_submodules "$nemoclaw_src"; then
is_routed_provider_requested && error "Failed to initialize the llm-router submodule required for routed inference."
warn "Submodule initialization failed — model router support may be unavailable"
fi
# Fetch version tags into the shallow clone so `git describe --tags
# --match "v*"` works at runtime (the shallow clone only has the
# single ref we asked for).
Expand All @@ -1418,7 +1371,6 @@ install_nemoclaw() {
spin "Installing ${_CLI_DISPLAY} dependencies" bash -c "cd \"$nemoclaw_src\" && npm install --ignore-scripts"
spin "Building ${_CLI_DISPLAY} CLI modules" bash -c "cd \"$nemoclaw_src\" && npm run --if-present build:cli"
spin "Building ${_CLI_DISPLAY} plugin" bash -c "cd \"$nemoclaw_src\"/nemoclaw && npm install --ignore-scripts && npm run build"
install_model_router_if_present "$nemoclaw_src"
spin "Linking ${_CLI_DISPLAY} CLI" bash -c "cd \"$nemoclaw_src\" && npm link"

# Install/upgrade the OpenShell CLI on the GitHub-clone path (curl|bash).
Expand Down
5 changes: 3 additions & 2 deletions src/lib/domain/installer/provider.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ describe("installer provider helpers", () => {
expect(normalizeInstallerProvider("anthropiccompatible")).toBe("anthropicCompatible");
expect(normalizeInstallerProvider(" AnthropicCompatible ")).toBe("anthropicCompatible");
expect(normalizeInstallerProvider("vllm")).toBe("vllm");
expect(normalizeInstallerProvider("routed")).toBe("routed");
expect(normalizeInstallerProvider("")).toBeNull();
expect(normalizeInstallerProvider("unsupported")).toBeNull();
});
Expand All @@ -33,11 +34,11 @@ describe("installer provider helpers", () => {

it("keeps help text values aligned with install.sh usage", () => {
expect(installerProviderHelpValues()).toBe(
"build, openai, anthropic, anthropicCompatible, gemini, ollama, custom, nim-local, vllm",
"build, openai, anthropic, anthropicCompatible, gemini, ollama, custom, nim-local, vllm, routed",
);
expect(installerProviderUsageLines()).toEqual([
"build | openai | anthropic | anthropicCompatible",
"gemini | ollama | custom | nim-local | vllm",
"gemini | ollama | custom | nim-local | vllm | routed",
"aliases: anthropiccompatible -> anthropicCompatible, cloud -> build, nim -> nim-local",
]);
});
Expand Down
1 change: 1 addition & 0 deletions src/lib/domain/installer/provider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ export const INSTALLER_PROVIDER_VALUES = [
"custom",
"nim-local",
"vllm",
"routed",
] as const;

export type InstallerProvider = (typeof INSTALLER_PROVIDER_VALUES)[number];
Expand Down
83 changes: 80 additions & 3 deletions src/lib/onboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -826,6 +826,8 @@ function loadBlueprintProfile(
const ROUTER_HEALTH_RETRIES = 15;
const ROUTER_HEALTH_INTERVAL_MS = 2000;
const ROUTER_HEALTH_TIMEOUT_MS = 3000;
const MODEL_ROUTER_RELATIVE_DIR = path.join("nemoclaw-blueprint", "router", "llm-router");
const MODEL_ROUTER_VENV_DIR = path.join(os.homedir(), ".nemoclaw", "model-router-venv");

async function isRouterHealthy(port: number, timeoutMs = ROUTER_HEALTH_TIMEOUT_MS): Promise<boolean> {
const http = require("http");
Expand Down Expand Up @@ -880,12 +882,87 @@ async function stopModelRouterProcess(pid: number, port: number): Promise<void>
}
}

function resolveHostCommandPath(commandName: string): string | null {
const result = runCapture(["sh", "-c", 'command -v "$1"', "--", commandName], {
ignoreError: true,
}).trim();
return result || null;
}

function modelRouterPackageDir(): string {
return path.join(ROOT, MODEL_ROUTER_RELATIVE_DIR);
}

function isModelRouterPackageReady(routerDir = modelRouterPackageDir()): boolean {
return fs.existsSync(path.join(routerDir, "pyproject.toml")) ||
fs.existsSync(path.join(routerDir, "setup.py"));
}

function initializeModelRouterSubmodule(routerDir = modelRouterPackageDir()): void {
if (isModelRouterPackageReady(routerDir)) return;
if (!fs.existsSync(path.join(ROOT, ".gitmodules")) || !fs.existsSync(path.join(ROOT, ".git"))) {
return;
}
console.log(" Initializing Model Router source...");
run(["git", "-C", ROOT, "submodule", "update", "--init", "--depth", "1", MODEL_ROUTER_RELATIVE_DIR], {
ignoreError: true,
});
}

function installModelRouterCommand(routerDir = modelRouterPackageDir()): string {
initializeModelRouterSubmodule(routerDir);
if (!isModelRouterPackageReady(routerDir)) {
throw new Error(
`Model Router source is not initialized at ${routerDir}. ` +
`Run: git -C ${ROOT} submodule update --init --depth 1 ${MODEL_ROUTER_RELATIVE_DIR}`,
);
}

if (!resolveHostCommandPath("python3")) {
throw new Error("python3 is required to prepare Model Router.");
}

const venvDir = process.env.NEMOCLAW_MODEL_ROUTER_VENV || MODEL_ROUTER_VENV_DIR;
const venvPython = path.join(venvDir, "bin", "python");
const routerCommand = path.join(venvDir, "bin", "model-router");

fs.mkdirSync(path.dirname(venvDir), { recursive: true });
console.log(` Preparing Model Router environment: ${venvDir}`);
const venvResult = run(["python3", "-m", "venv", venvDir], {
ignoreError: true,
timeout: 120_000,
});
if (venvResult.status !== 0 || !fs.existsSync(venvPython)) {
throw new Error("Failed to create Model Router virtual environment.");
}

const installResult = run(
[venvPython, "-m", "pip", "install", "--quiet", "--upgrade", `${routerDir}[prefill,proxy]`],
{
ignoreError: true,
timeout: 600_000,
},
);
if (installResult.status !== 0) {
throw new Error("Failed to install Model Router dependencies.");
}
if (!fs.existsSync(routerCommand)) {
throw new Error("Model Router install did not produce the model-router command.");
}
return routerCommand;
}

function ensureModelRouterCommand(): string {
return resolveHostCommandPath("model-router") || installModelRouterCommand();
Comment thread
ericksoa marked this conversation as resolved.
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.

/**
* Start the model-router proxy and wait for it to become healthy.
* Follows the same pattern as Ollama startup (spawn detached, poll health).
* Returns the PID of the child process.
*/
async function startModelRouter(routerCfg: BlueprintRouterConfig): Promise<number> {
const routerCommand = ensureModelRouterCommand();
const port = routerCfg.port || 4000;
const blueprintDir = path.join(ROOT, "nemoclaw-blueprint");
const poolConfigPath = path.join(
Expand All @@ -898,7 +975,7 @@ async function startModelRouter(routerCfg: BlueprintRouterConfig): Promise<numbe
fs.mkdirSync(stateDir, { recursive: true });

const proxyConfigResult = spawnSync(
"model-router",
routerCommand,
["proxy-config", "--config", poolConfigPath, "--output", litellmConfigPath],
{ encoding: "utf8", timeout: 30_000, cwd: blueprintDir },
);
Expand Down Expand Up @@ -931,7 +1008,7 @@ async function startModelRouter(routerCfg: BlueprintRouterConfig): Promise<numbe
}

const child = spawn(
"model-router",
routerCommand,
[
"proxy",
"--litellm-config", litellmConfigPath,
Expand Down Expand Up @@ -5905,7 +5982,7 @@ async function setupNim(
// Model Router: complexity-based routing via blueprint config.
const blueprintRouterCfg = loadBlueprintProfile("routed");
if (blueprintRouterCfg && blueprintRouterCfg.router?.enabled === true) {
options.push({ key: "routed", label: "Model Router (complexity-based routing)" });
options.push({ key: "routed", label: "Model Router (experimental)" });
}

function checkOllamaPortsOrWarn(): boolean {
Expand Down
40 changes: 38 additions & 2 deletions test/install-preflight.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -488,7 +488,7 @@ exit 98
expect(output).toMatch(/--version/);
expect(output).toMatch(/NEMOCLAW_PROVIDER/);
expect(output).toMatch(/build \| openai \| anthropic \| anthropicCompatible/);
expect(output).toMatch(/gemini \| ollama \| custom \| nim-local \| vllm/);
expect(output).toMatch(/gemini \| ollama \| custom \| nim-local \| vllm \| routed/);
expect(output).toMatch(/aliases: cloud -> build, nim -> nim-local/);
expect(output).toMatch(/NEMOCLAW_POLICY_MODE/);
expect(output).toMatch(/NEMOCLAW_SANDBOX_NAME/);
Expand All @@ -504,7 +504,7 @@ exit 98
const output = `${result.stdout}${result.stderr}`;
expect(result.status).toBe(0);
expect(output).toMatch(/build \| openai \| anthropic \| anthropicCompatible/);
expect(output).toMatch(/gemini \| ollama \| custom \| nim-local \| vllm/);
expect(output).toMatch(/gemini \| ollama \| custom \| nim-local \| vllm \| routed/);
expect(output).toMatch(/aliases: cloud -> build, nim -> nim-local/);
});

Expand Down Expand Up @@ -563,11 +563,34 @@ exit 98
const fakeBin = path.join(tmp, "bin");
const prefix = path.join(tmp, "prefix");
const npmLog = path.join(tmp, "npm.log");
const pythonLog = path.join(tmp, "python.log");
const gitLog = path.join(tmp, "git.log");
fs.mkdirSync(fakeBin);
fs.mkdirSync(path.join(tmp, ".git"));
fs.mkdirSync(path.join(prefix, "bin"), { recursive: true });

writeNodeStub(fakeBin);
writeExecutable(
path.join(fakeBin, "git"),
`#!/usr/bin/env bash
printf 'git %s\\n' "$*" >> "$GIT_LOG_PATH"
exit 90
`,
);
writeExecutable(
path.join(fakeBin, "python3"),
`#!/usr/bin/env bash
printf 'python3 %s\\n' "$*" >> "$PYTHON_LOG_PATH"
exit 88
`,
);
writeExecutable(
path.join(fakeBin, "pip3"),
`#!/usr/bin/env bash
printf 'pip3 %s\\n' "$*" >> "$PYTHON_LOG_PATH"
exit 89
`,
);
writeNpmStub(
fakeBin,
`printf '%s\\n' "$*" >> "$NPM_LOG_PATH"
Expand Down Expand Up @@ -602,6 +625,13 @@ fi`,
path.join(tmp, "nemoclaw", "package.json"),
JSON.stringify({ name: "nemoclaw-plugin", version: "0.1.0" }, null, 2),
);
fs.mkdirSync(path.join(tmp, "nemoclaw-blueprint", "router", "llm-router"), {
recursive: true,
});
fs.writeFileSync(
path.join(tmp, "nemoclaw-blueprint", "router", "llm-router", "pyproject.toml"),
"[project]\nname = 'llm-router'\n",
);

const result = spawnSync("bash", [INSTALLER], {
cwd: tmp,
Expand All @@ -614,6 +644,8 @@ fi`,
NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1",
NPM_PREFIX: prefix,
NPM_LOG_PATH: npmLog,
PYTHON_LOG_PATH: pythonLog,
GIT_LOG_PATH: gitLog,
},
});

Expand All @@ -624,6 +656,10 @@ fi`,
expect(log).toMatch(/^link/m);
// the GitHub URL must NOT appear — this is a local install
expect(log).not.toMatch(new RegExp(GITHUB_INSTALL_URL.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")));
// Model Router must not run provider-specific dependency setup from the generic installer.
expect(fs.existsSync(pythonLog)).toBe(false);
const gitCalls = fs.existsSync(gitLog) ? fs.readFileSync(gitLog, "utf-8") : "";
expect(gitCalls).not.toMatch(/submodule/);
});

it("auto-resumes an interrupted onboarding session during install", () => {
Expand Down
Loading
Loading