Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/inference/switch-models.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ For a compatible endpoint, omit `--endpoint-url` when the durable registry entry
NemoClaw reuses the recorded route and does not repoint the gateway.
If the route metadata is incomplete, NemoClaw stops and tells you to re-run onboarding.

For Hermes, the command also mirrors the selected model into the dashboard profile.
If it reports that the Dashboard config did not converge, the route and main Hermes config remain committed; follow [Hermes dashboard config did not converge](../../reference/troubleshooting#hermes-dashboard-config-did-not-converge) before using Dashboard Chat.

</AgentOnly>

<AgentOnly variant="deepagents">
Expand Down
5 changes: 4 additions & 1 deletion docs/inference/switch-providers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,10 @@ Changes within the current API family hot-reload without replacing the gateway p
When the API family changes, NemoClaw commits the route and configuration, then restarts only the OpenClaw gateway and verifies its health.

For Hermes, NemoClaw updates `/sandbox/.hermes/config.yaml`, including the model, base URL, API-family mode, and OpenShell proxy API-key placeholder.
Hermes does not rebuild or restart for this runtime route change.
When the dashboard profile exists, NemoClaw also mirrors the route into `/sandbox/.hermes/dashboard-home/config.yaml`; a normal runtime route change does not rebuild or restart Hermes.
If that dashboard mirror cannot be confirmed, the route and main config remain committed but the command exits nonzero.
Follow [Hermes dashboard config did not converge](../../reference/troubleshooting#hermes-dashboard-config-did-not-converge) before using Dashboard Chat.
A missing dashboard profile is treated as disabled and does not fail the switch.

If the in-sandbox configuration sync fails after the gateway route changes, NemoClaw keeps the gateway and host registry aligned and prints a rebuild hint.
Run the rebuild before relying on the running agent.
Expand Down
4 changes: 4 additions & 0 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2844,7 +2844,11 @@ Switch the active inference provider or model for a NemoClaw-managed Hermes sand
The command updates the OpenShell gateway route, patches the selected running agent config so it matches the route, recomputes the config hash, and updates the NemoClaw registry.
It is also available in sandbox-first form as `$$nemoclaw <name> inference set --provider <provider> --model <model>`.
For Hermes, the patch updates `/sandbox/.hermes/config.yaml` (`model.default`, `model.base_url`, `model.provider: custom`, API-family mode when needed, and the OpenShell proxy API-key placeholder) and does not rebuild or restart the gateway.
When the Hermes dashboard profile exists, the command also mirrors the model route into `/sandbox/.hermes/dashboard-home/config.yaml` for Dashboard Chat.
Keeping the placeholder preserves dashboard and API authentication after provider switches.
If NemoClaw cannot confirm that the dashboard config was updated, the route, registry, and main Hermes config remain committed, but the command exits nonzero without printing `Inference route synced`.
Restart the sandbox with `nemohermes <name> stop` followed by `nemohermes <name> start`, then verify Dashboard Chat before relying on it.
A missing dashboard profile is treated as disabled and does not fail the switch.
Under the `nemohermes` alias, it uses the registered Hermes sandbox when exactly one exists; otherwise pass `--sandbox <name>` to target one explicitly.

</AgentOnly>
Expand Down
16 changes: 16 additions & 0 deletions docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2574,6 +2574,22 @@ After the rebuild completes, return to the Skills page to confirm the skill is r
The issues below are common problems you may encounter when running Hermes through `nemohermes`.
For setup, refer to [Quickstart with Hermes](../../hermes/get-started/quickstart).

### Hermes dashboard config did not converge

`nemohermes inference set` updates the OpenShell route, registry, and `/sandbox/.hermes/config.yaml` before it refreshes the separate dashboard profile.
If the dashboard profile exists but NemoClaw cannot confirm that `/sandbox/.hermes/dashboard-home/config.yaml` was updated, the command exits nonzero without printing `Inference route synced`.
The committed route and main Hermes config are not rolled back.

Restart the sandbox so startup mirrors the committed model route into the dashboard profile:

```bash
nemohermes <name> stop
nemohermes <name> start
```

Then run `nemohermes inference get` and verify Dashboard Chat uses the selected model.
If the command succeeds because the dashboard profile is missing, the dashboard is disabled and no dashboard recovery is required.

### Hermes restart reports `config hash mismatch`

A Hermes restart reports `config hash mismatch` when a strict root-owned hash is available and `/sandbox/.hermes/config.yaml` or `/sandbox/.hermes/.env` does not match it.
Expand Down
13 changes: 12 additions & 1 deletion src/lib/actions/inference-set-gateway-restart.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,14 @@ interface InferenceResultForGateway {
model: string;
primaryModelRef: string;
inSandboxConfigSynced: boolean;
/**
* Hermes only: whether the isolated Web Dashboard profile converged onto the
* switched model (#6893). `undefined` for agents/switches with no Dashboard to
* converge (treated as converged). When explicitly `false` the "Inference route
* synced" line is withheld and the caller raises a post-commit failure so the
* command cannot claim a route it did not fully apply.
*/
dashboardConverged?: boolean;
}

export interface InferenceMutation<T extends InferenceResultForGateway> {
Expand Down Expand Up @@ -101,7 +109,10 @@ export function finalizeInferenceMutation<T extends InferenceResultForGateway>(
deps.appendAuditEntry(auditEntry);
}

if (result.inSandboxConfigSynced && !openClawGatewayRestartRequired) {
// A Hermes switch whose Web Dashboard profile did not converge is not fully
// applied, so withhold the success line (the caller already warned) (#6893).
const hermesDashboardStale = agentName === "hermes" && result.dashboardConverged === false;
if (result.inSandboxConfigSynced && !openClawGatewayRestartRequired && !hermesDashboardStale) {
deps.log(
agentName === "hermes"
? ` Inference route synced for '${result.sandboxName}': ${result.model}`
Expand Down
199 changes: 199 additions & 0 deletions src/lib/actions/inference-set-hermes-run.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,205 @@ describe("runInferenceSet Hermes routing", () => {
expect(deps.calls.restartSandboxGateway).not.toHaveBeenCalled();
});

it("re-seeds the isolated Hermes dashboard config after an in-place switch (#6893)", async () => {
const config: ConfigObject = {
model: {
default: "moonshotai/kimi-k2.6",
provider: "custom",
base_url: "https://inference.local/v1",
},
};
const deps = createDeps({
config,
entry: {
name: "hermes",
agent: "hermes",
provider: "hermes-provider",
model: "moonshotai/kimi-k2.6",
},
defaultSandbox: "hermes",
target: HERMES_TARGET,
session: baseSession({ agent: "hermes", sandboxName: "hermes" }),
});

await runInferenceSet(
{
provider: "hermes-provider",
model: "openai/gpt-5.4-mini",
sandboxName: "hermes",
noVerify: true,
},
deps,
);

// The dashboard-home config only re-mirrors the gateway model routing at
// startup, so the in-place switch must re-seed it or Dashboard Chat stays on
// the previous model. It must run after the gateway config was written.
expect(deps.calls.seedHermesDashboardConfig).toHaveBeenCalledWith("hermes", HERMES_TARGET);
const writeOrder = deps.calls.writeSandboxConfig.mock.invocationCallOrder[0];
const seedOrder = deps.calls.seedHermesDashboardConfig.mock.invocationCallOrder[0];
expect(seedOrder).toBeGreaterThan(writeOrder);
});

it("does not re-seed the dashboard when the in-sandbox config write fails (#6893)", async () => {
const config: ConfigObject = {
model: { default: "moonshotai/kimi-k2.6", provider: "custom" },
};
const deps = createDeps({
config,
entry: {
name: "hermes",
agent: "hermes",
provider: "hermes-provider",
model: "moonshotai/kimi-k2.6",
},
defaultSandbox: "hermes",
target: HERMES_TARGET,
session: baseSession({ agent: "hermes", sandboxName: "hermes" }),
});
deps.calls.writeSandboxConfig.mockImplementation(() => {
throw new Error("write failed");
});

await runInferenceSet(
{
provider: "hermes-provider",
model: "openai/gpt-5.4-mini",
sandboxName: "hermes",
noVerify: true,
},
deps,
);

// A failed gateway-config write leaves the old config in place; re-seeding the
// dashboard from it would be pointless (and the guidance is to rebuild).
expect(deps.calls.seedHermesDashboardConfig).not.toHaveBeenCalled();
});

it("does not re-seed or report synced when the config hash refresh fails (#6893)", async () => {
const config: ConfigObject = {
model: { default: "moonshotai/kimi-k2.6", provider: "custom" },
};
const deps = createDeps({
config,
entry: {
name: "hermes",
agent: "hermes",
provider: "hermes-provider",
model: "moonshotai/kimi-k2.6",
},
defaultSandbox: "hermes",
target: HERMES_TARGET,
session: baseSession({ agent: "hermes", sandboxName: "hermes" }),
});
deps.calls.recomputeSandboxConfigHash.mockImplementation(() => {
throw new Error("hash refresh failed");
});

await runInferenceSet(
{
provider: "hermes-provider",
model: "openai/gpt-5.4-mini",
sandboxName: "hermes",
noVerify: true,
},
deps,
);

expect(deps.calls.writeSandboxConfig).toHaveBeenCalledOnce();
expect(deps.calls.seedHermesDashboardConfig).not.toHaveBeenCalled();
const logs = deps.calls.log.mock.calls.map((call) => String(call[0]));
expect(logs.some((line) => line.includes("failed to refresh its integrity hash"))).toBe(true);
expect(logs.some((line) => line.includes("rebuild"))).toBe(true);
expect(logs.some((line) => line.includes("Inference route synced"))).toBe(false);
});

it("fails after commit when the dashboard does not converge (#6893)", async () => {
const config: ConfigObject = {
model: {
default: "moonshotai/kimi-k2.6",
provider: "custom",
base_url: "https://inference.local/v1",
},
};
const deps = createDeps({
config,
entry: {
name: "hermes",
agent: "hermes",
provider: "hermes-provider",
model: "moonshotai/kimi-k2.6",
},
defaultSandbox: "hermes",
target: HERMES_TARGET,
session: baseSession({ agent: "hermes", sandboxName: "hermes" }),
seedHermesDashboardConfigResult: "failed",
});

await expect(
runInferenceSet(
{
provider: "hermes-provider",
model: "openai/gpt-5.4-mini",
sandboxName: "hermes",
noVerify: true,
},
deps,
),
).rejects.toMatchObject({
name: "InferenceSetError",
exitCode: 1,
message: expect.stringMatching(/committed route was not rolled back.*Restart the sandbox/u),
});

// The route, main config, durable session, and audit are already committed,
// but the command must fail instead of claiming complete convergence.
expect(deps.getSession()?.model).toBe("openai/gpt-5.4-mini");
expect(deps.calls.appendAuditEntry).toHaveBeenCalledOnce();
const logs = deps.calls.log.mock.calls.map((c) => String(c[0]));
expect(logs.some((l) => l.includes("Inference route synced"))).toBe(false);
expect(logs.some((l) => l.includes("could not refresh the dashboard"))).toBe(true);
});

it("still reports synced when the dashboard profile is absent (Dashboard disabled) (#6893)", async () => {
const config: ConfigObject = {
model: {
default: "moonshotai/kimi-k2.6",
provider: "custom",
base_url: "https://inference.local/v1",
},
};
const deps = createDeps({
config,
entry: {
name: "hermes",
agent: "hermes",
provider: "hermes-provider",
model: "moonshotai/kimi-k2.6",
},
defaultSandbox: "hermes",
target: HERMES_TARGET,
session: baseSession({ agent: "hermes", sandboxName: "hermes" }),
seedHermesDashboardConfigResult: "absent",
});

await runInferenceSet(
{
provider: "hermes-provider",
model: "openai/gpt-5.4-mini",
sandboxName: "hermes",
noVerify: true,
},
deps,
);

// Nothing to converge — the switch is fully applied, so it still reports synced
// and does not warn.
const logs = deps.calls.log.mock.calls.map((c) => String(c[0]));
expect(logs.some((l) => l.includes("Inference route synced"))).toBe(true);
expect(logs.some((l) => l.includes("could not refresh the dashboard"))).toBe(false);
});

it("keeps Hermes custom Anthropic switches off the managed Anthropic SSE frontend (#6289)", async () => {
const config: ConfigObject = {
model: {
Expand Down
2 changes: 2 additions & 0 deletions src/lib/actions/inference-set-openclaw-run.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ describe("runInferenceSet OpenClaw routing", () => {
});
expect(deps.calls.writeSandboxConfig).toHaveBeenCalledWith("alpha", OPENCLAW_TARGET, config);
expect(deps.calls.recomputeSandboxConfigHash).toHaveBeenCalledWith("alpha", OPENCLAW_TARGET);
// The dashboard re-seed is Hermes-only; OpenClaw has no isolated dashboard config. (#6893)
expect(deps.calls.seedHermesDashboardConfig).not.toHaveBeenCalled();
expect(deps.calls.updateSandbox).toHaveBeenCalledWith(
"alpha",
expect.objectContaining({
Expand Down
4 changes: 4 additions & 0 deletions src/lib/actions/inference-set.test-support.ts
Original file line number Diff line number Diff line change
Expand Up @@ -86,12 +86,14 @@ export function createDeps(options: {
prepareRunOpenshell?: () => void;
rewriteConfigUrlsWithDnsPinning?: (value: ConfigValue) => Promise<ConfigValue>;
restartSandboxGateway?: InferenceSetDeps["restartSandboxGateway"];
seedHermesDashboardConfigResult?: "converged" | "absent" | "failed";
withGatewayRouteMutationLock?: InferenceSetDeps["withGatewayRouteMutationLock"];
}): InferenceSetDeps & {
calls: {
captureOpenshell: ReturnType<typeof vi.fn>;
writeSandboxConfig: ReturnType<typeof vi.fn>;
recomputeSandboxConfigHash: ReturnType<typeof vi.fn>;
seedHermesDashboardConfig: ReturnType<typeof vi.fn>;
updateSandbox: ReturnType<typeof vi.fn>;
readSandboxConfig: ReturnType<typeof vi.fn>;
updateSession: ReturnType<typeof vi.fn>;
Expand Down Expand Up @@ -124,6 +126,7 @@ export function createDeps(options: {
})),
writeSandboxConfig: vi.fn(),
recomputeSandboxConfigHash: vi.fn(),
seedHermesDashboardConfig: vi.fn(() => options.seedHermesDashboardConfigResult ?? "converged"),
updateSandbox: vi.fn(() => true),
readSandboxConfig: vi.fn(() => options.config),
updateSession: vi.fn((mutator: (value: Session) => Session | void) => {
Expand Down Expand Up @@ -169,6 +172,7 @@ export function createDeps(options: {
readSandboxConfig: calls.readSandboxConfig,
writeSandboxConfig: calls.writeSandboxConfig,
recomputeSandboxConfigHash: calls.recomputeSandboxConfigHash,
seedHermesDashboardConfig: calls.seedHermesDashboardConfig,
prepareRunOpenshell: calls.prepareRunOpenshell,
captureOpenshell: calls.captureOpenshell,
appendAuditEntry: calls.appendAuditEntry,
Expand Down
Loading
Loading