Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 100 additions & 2 deletions docs/inference/set-up-openai-compatible-endpoint.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
title: "Set Up an OpenAI-Compatible Endpoint"
sidebar-title: "OpenAI-Compatible Endpoint"
description: "Connect NemoClaw to a self-hosted or custom OpenAI-compatible inference endpoint."
description-agent: "Shows how to configure an OpenAI-compatible endpoint for NemoClaw, including raw model files and non-interactive onboarding."
keywords: ["nemoclaw openai compatible endpoint", "custom inference endpoint", "self-hosted inference"]
description-agent: "Shows how to configure an OpenAI-compatible endpoint for NemoClaw, including portable inference descriptors, raw model files, and non-interactive onboarding."
keywords: ["nemoclaw openai compatible endpoint", "portable inference descriptor", "custom inference endpoint", "self-hosted inference"]
content:
type: "how_to"
---
Expand Down Expand Up @@ -64,6 +64,104 @@ Other URLs still require `COMPATIBLE_API_KEY`.

Refer to [Choose a Compatible Inference API](choose-compatible-inference-api) for the probe order and runtime API selection.

## Supply a Portable Inference Descriptor

A host-side activation component can select a compatible endpoint for the portable experimental profile.
The component must write the descriptor before the installer or onboarding command starts.
The activation component owns authentication to the descriptor source and writes only the five resolved inference fields.

<Warning>
The descriptor contains an API key.
Use a real `/run/nemoclaw` directory owned by root or the user who runs NemoClaw.
Do not permit group or other users to write that directory.
Publish the descriptor only at `/run/nemoclaw/portable-inference.json`.
Use a regular file with one hard link, mode `0600`, and ownership by the user who runs NemoClaw.
Do not add descriptor-source locations or source credentials to the descriptor.
Do not include the descriptor or its values in a repository, image, log, shell argument, artifact, or activation-component persistent state.
Use a short-lived API key and set `expiresAt` to that credential's expiration time.
</Warning>

Create a temporary regular file in `/run/nemoclaw` with the required owner and mode `0600`.
Write the complete JSON.
Close the temporary file.
Rename that file to `/run/nemoclaw/portable-inference.json`.
This atomic replacement prevents NemoClaw from reading a partial descriptor.
The descriptor uses this schema:

```json
{
"schemaVersion": 1,
"apiKey": "<short-lived-api-key>",
"baseUrl": "https://inference.example.com/v1",
"model": "example-model",
"expiresAt": "<future-ISO-8601-UTC-timestamp>"
}
```

Each field has one required purpose:

| Field | Requirement |
|---|---|
| `schemaVersion` | Use the integer `1`. |
| `apiKey` | Supply the short-lived API key for the compatible endpoint. |
| `baseUrl` | Supply a compatible endpoint base URL that uses HTTPS, without credentials, a query, or a fragment. NemoClaw applies its existing endpoint and server-side request forgery (SSRF) policy. |
| `model` | Supply the provider model ID. |
| `expiresAt` | Supply a future ISO 8601 UTC timestamp that matches the API key lifetime. |

Run the portable installer after the final descriptor is available:

<AgentOnly variant="openclaw">

```bash
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash -s -- --experimental-profile portable --fresh
```

</AgentOnly>
<AgentOnly variant="hermes">

```bash
curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_AGENT=hermes bash -s -- --experimental-profile portable --fresh
```

</AgentOnly>
<AgentOnly variant="deepagents">

```bash
curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_AGENT=langchain-deepagents-code bash -s -- --experimental-profile portable --fresh
```

</AgentOnly>

The portable profile handles the descriptor as follows:

| Descriptor state | Onboarding result |
|---|---|
| Absent | NemoClaw keeps the existing portable behavior. The local Podman `qwen3-vl:4b` model remains the active inference route. |
| Directory and file metadata meet the requirements above, the descriptor is valid, and the authenticated onboarding checks pass | NemoClaw makes the compatible endpoint model the active inference route. If the local Podman `qwen3-vl:4b` runner already exists, NemoClaw leaves it installed as standby. |
| File passes the metadata checks but contains malformed JSON, an invalid schema, an expired credential, or a rejected endpoint | NemoClaw deletes the descriptor and exits before it changes gateway, provider, sandbox, or onboarding state. |
| Descriptor is valid, but the endpoint, selected model, or configured route fails an authenticated onboarding check | NemoClaw deletes the descriptor and exits without reporting onboarding success. The compatible route may already be configured. NemoClaw does not activate an existing local runner automatically. |
| Descriptor entry is present, but directory or file metadata does not meet the requirements above | NemoClaw does not read or delete the filesystem entry. It exits before onboarding changes state. The activation component or operator must atomically replace the entry. |

OpenShell keeps one active inference route.
The local runner is standby only when it already exists.
OpenShell does not automatically switch to that runner when the compatible endpoint is unavailable.
Use [Switch Inference Providers](../manage-inference/switch-providers) when you need to change the active route.

Endpoint validation is a point-in-time onboarding check, not continuous health monitoring.
During onboarding, NemoClaw sends an authenticated Chat Completions request for the selected model.
After route setup, applicable flows also verify that the sandbox receives non-empty assistant content through `inference.local`.
These checks do not provide continuous availability monitoring or automatic failover.

NemoClaw consumes and deletes a descriptor only after its file metadata passes these checks.
It deletes an admitted descriptor whether it accepts or rejects the descriptor content.
During onboarding, NemoClaw holds the API key in an asynchronous in-process credential scope instead of `process.env`.
Unrelated child processes do not inherit the API key.
Compatible-endpoint validation reads the scoped value, and provider registration passes it explicitly to OpenShell.
After registration, OpenShell holds the provider credential and adds it to managed inference requests.
NemoClaw does not write the API key into the sandbox or its persistent state.
The upstream credential expiration still controls the registered credential's lifetime.
The activation component must supply a new descriptor for each onboarding attempt that needs the compatible endpoint.

## Serve a Raw Model File

Start a compatible server for a raw model file instead of passing the file path to NemoClaw.
Expand Down
27 changes: 27 additions & 0 deletions src/lib/credentials/scoped-overrides.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import { AsyncLocalStorage } from "node:async_hooks";

const scopedCredentialOverrides = new AsyncLocalStorage<ReadonlyMap<string, string>>();

/** Make validated credential overrides available to one asynchronous call tree. */
export async function withCredentialOverrides<T>(
values: Readonly<Record<string, string>>,
operation: () => Promise<T> | T,
): Promise<T> {
const inherited = scopedCredentialOverrides.getStore();
const overrides = new Map(inherited ?? []);
for (const [key, rawValue] of Object.entries(values)) {
if (/[\u0000\r\n]/u.test(rawValue)) {
throw new Error(`Scoped credential '${key}' must not contain NUL, CR, or LF.`);
}
if (!rawValue.trim()) throw new Error(`Scoped credential '${key}' must not be empty.`);
overrides.set(key, rawValue);
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
return scopedCredentialOverrides.run(overrides, operation);
}

export function getScopedCredentialOverride(key: string): string | null {
return scopedCredentialOverrides.getStore()?.get(key) ?? null;
}
24 changes: 16 additions & 8 deletions src/lib/credentials/store.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
// Host-side credential helpers.
//
// The OpenShell gateway is the system of record for provider credentials.
// This module holds them only in the current process environment so they
// can be passed through to `openshell provider create/update --credential KEY`
// during onboarding. Nothing is written to disk.
// This module exposes staged process-environment values and asynchronous
// in-process overrides. Callers pass the selected value explicitly to
// `openshell provider create/update --credential KEY`. Nothing is written to disk.

import fs from "node:fs";
import os from "node:os";
Expand All @@ -19,6 +19,9 @@ import { createPromptActivityCleanup } from "../core/prompt-activity";
import { listMessagingCredentialMetadata } from "../messaging/channels";
import { rejectSymlinksOnPath } from "../state/config-io";
import { nemoclawStateRoot } from "../state/state-root";
import { getScopedCredentialOverride } from "./scoped-overrides";

export { withCredentialOverrides } from "./scoped-overrides";

const UNSAFE_HOME_PATHS = new Set(["/tmp", "/var/tmp", "/dev/shm", "/"]);

Expand Down Expand Up @@ -182,8 +185,10 @@ export function saveCredential(key: string, value: CredentialInput): void {
}
}

/** Return the staged value for `key` from the current process env, or null. */
/** Return the scoped or staged value for `key`, or null. */
export function getCredential(key: string): string | null {
const scoped = getScopedCredentialOverride(key);
if (scoped) return scoped;
const raw = process.env[key];
if (!raw) return null;
const normalized = normalizeCredentialValue(raw);
Expand All @@ -200,10 +205,11 @@ function getLegacyCredentialAlias(envName: string): string | null {

/**
* Canonical entry point for provider credential resolution (PR #2306).
* Resolves the credential for `envName` from `process.env`, falling back
* to a one-time on-demand stage of any pre-fix `~/.nemoclaw/credentials.json`,
* and writes the resolved value back into `process.env` so downstream
* code that reads `process.env[envName]` directly sees it.
* Resolves an asynchronous in-process override before `process.env`.
* Without an override, falls back to a one-time on-demand stage of any
* pre-fix `~/.nemoclaw/credentials.json` and writes the resolved value back
* into `process.env` for downstream compatibility. A scoped override never
* enters `process.env` through this function.
*
* Returns the resolved value, or `null` if neither env nor the legacy
* file produced one.
Expand All @@ -218,6 +224,8 @@ function getLegacyCredentialAlias(envName: string): string | null {
* guard inside the staging helper itself.
*/
export function resolveProviderCredential(envName: string): string | null {
const scoped = getScopedCredentialOverride(envName);
if (scoped) return scoped;
let value = getCredential(envName) || getLegacyCredentialAlias(envName);
if (!value) {
stageLegacyCredentialsToEnv();
Expand Down
70 changes: 69 additions & 1 deletion src/lib/onboard/command.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,12 @@ import path from "node:path";

import { afterEach, describe, expect, it, vi } from "vitest";

import { getCredential } from "../credentials/store";
import { loadServingCatalog } from "../inference/serving/catalog-loader";
import { servingProfileProvenance } from "../inference/serving/profile-provenance";
import { resolveOnboardOptions, runOnboardCommand } from "./command";
import type { OnboardFlags } from "./command-support";
import { PortableInferenceDescriptorError } from "./experimental/portable-inference-descriptor";
import { invalidGatewayManagementDeclarationError } from "./gateway-management";
import { GatewayAuthorityError } from "./gateway-teardown-authority";
import {
Expand Down Expand Up @@ -248,6 +250,7 @@ describe("onboard command options", () => {
autoYes: true,
noOllamaAutostart: true,
experimentalProfile: null,
portableInferenceActivation: null,
servingProfile: null,
servingProfileProvenance: null,
});
Expand Down Expand Up @@ -276,6 +279,7 @@ describe("onboard command options", () => {
autoYes: false,
noOllamaAutostart: false,
experimentalProfile: null,
portableInferenceActivation: null,
servingProfile: null,
servingProfileProvenance: null,
});
Expand Down Expand Up @@ -488,7 +492,7 @@ describe("onboard command options", () => {
expect(env.NEMOCLAW_SERVING_PRESET).toBeUndefined();
});

it("prepares and scopes portable profile defaults around onboarding", async () => {
it("keeps local portable defaults when no descriptor is present", async () => {
const env: NodeJS.ProcessEnv = {
NEMOCLAW_EXPERIMENTAL_PROFILE: "previous-profile",
NEMOCLAW_PROVIDER: "previous-provider",
Expand All @@ -502,6 +506,7 @@ describe("onboard command options", () => {
await runOnboardCommand({
flags: { "experimental-profile": "portable" },
env,
loadPortableInferenceDescriptor: async () => null,
runOnboard: async () => {
for (const key of [
"NEMOCLAW_EXPERIMENTAL_PROFILE",
Expand Down Expand Up @@ -537,6 +542,69 @@ describe("onboard command options", () => {
});
});

it("configures portable onboarding from an admitted descriptor without exporting its credential", async () => {
vi.stubEnv("COMPATIBLE_API_KEY", undefined);
const env: NodeJS.ProcessEnv = {};
const runOnboard = vi.fn(async (options) => {
expect(options.portableInferenceActivation).toEqual({
schemaVersion: 1,
baseUrl: "https://inference.example.test/v1",
model: "vendor/model-1",
expiresAt: "2026-08-10T18:05:00Z",
});
expect(env).toMatchObject({
NEMOCLAW_PROVIDER: "custom",
NEMOCLAW_MODEL: "vendor/model-1",
NEMOCLAW_ENDPOINT_URL: "https://inference.example.test/v1",
NEMOCLAW_PREFERRED_API: "openai-completions",
});
expect(env.COMPATIBLE_API_KEY).toBeUndefined();
expect(process.env.COMPATIBLE_API_KEY).toBeUndefined();
expect(getCredential("COMPATIBLE_API_KEY")).toBe("runtime-only-secret");
});

await runOnboardCommand({
flags: { "experimental-profile": "portable" },
env,
loadPortableInferenceDescriptor: async () => ({
schemaVersion: 1,
apiKey: "runtime-only-secret",
baseUrl: "https://inference.example.test/v1",
model: "vendor/model-1",
expiresAt: "2026-08-10T18:05:00Z",
}),
runOnboard,
});

expect(runOnboard).toHaveBeenCalledOnce();
expect(env.NEMOCLAW_PROVIDER).toBeUndefined();
expect(env.NEMOCLAW_ENDPOINT_URL).toBeUndefined();
expect(getCredential("COMPATIBLE_API_KEY")).toBeNull();
});

it("fails before onboarding when a present portable descriptor is invalid", async () => {
const env: NodeJS.ProcessEnv = {};
const errors: string[] = [];
const runOnboard = vi.fn(async () => {});

await expect(
runOnboardCommand({
flags: { "experimental-profile": "portable" },
env,
loadPortableInferenceDescriptor: async () => {
throw new PortableInferenceDescriptorError("Runtime inference descriptor has expired.");
},
runOnboard,
error: (message = "") => errors.push(message),
exit: exitWithCode,
}),
).rejects.toThrow("exit:1");

expect(runOnboard).not.toHaveBeenCalled();
expect(env).toEqual({});
expect(errors).toEqual([" Runtime inference descriptor has expired."]);
});

it("scopes an agents manifest to one onboarding run", async () => {
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-agents-manifest-"));
const manifestPath = path.join(tmpDir, "agents.yaml");
Expand Down
Loading
Loading