Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 2 additions & 8 deletions docs/inference/set-up-vllm-on-two-dgx-sparks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -169,13 +169,6 @@ The explicit target still must match the exact reciprocal rail topology.

## Understand the Distributed Runtime

The compiled managed inference catalog reads cluster size, profile selection, artifacts, runtime resources, serving arguments, preparation, and readiness checks from preset and recipe YAML.
Maintainers can add another compatible profile that uses the registered managed-cluster adapters by adding preset and recipe YAML.
The YAML `nodeCount` declares the required cardinality.
The TypeScript adapters validate the qualified topology against it and materialize every ranked topology node.
The build compiles those YAML definitions into the packaged runtime catalog automatically.
A profile that needs a different execution contract requires a registered adapter.

The compiled profile pins these artifacts:

- The image is `ghcr.io/anemll/dspark-vllm-gx10@sha256:a83948492cf13df455170fb42885f5ef4db54fefe0feff0f841ecbff464ac9d8`.
Expand Down Expand Up @@ -234,7 +227,8 @@ $$nemoclaw uninstall

Full uninstall revalidates every exact container ID before it removes the containers.
It then retires the receipt, copied SSH bindings, and exact temporary discovery claims recorded by that receipt.
If revalidation, removal, or safe claim retirement fails, uninstall stops before other cleanup and preserves the receipt for a retry.
If NemoClaw cannot revalidate or remove a receipt-owned container, uninstall stops before other cleanup and preserves the receipt so you can rerun `$$nemoclaw uninstall`.
Uninstall also stops and preserves the receipt when NemoClaw cannot retire an exact temporary discovery claim recorded by that receipt.
A gateway-scoped uninstall preserves the host-global cluster, receipt, SSH bindings, and API key while another gateway remains.
Cluster cleanup does not delete the Hugging Face model cache.

Expand Down
19 changes: 11 additions & 8 deletions docs/manage-sandboxes/backup-restore.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -102,14 +102,17 @@ NemoClaw computes versions (`v1`, `v2`, through `vN`) from timestamp order, so `
`snapshot create` requires shields to be down.
Snapshot creation and restore share the per-sandbox transition lock with the shields auto-restore timer.

If a timed shields-down window expires during snapshot work, auto-restore closes the per-sandbox lifecycle deadline gate.
The gate blocks new mutations and waits for the recorded live owner to release its exact lock generation.
NemoClaw does not signal the snapshot process.
If ownership becomes ambiguous, NemoClaw attempts to record durable containment and reports exact-generation recovery guidance.
If the containment commit fails, NemoClaw retains any exact lifecycle and deadline gates it already owns.
A state-directory failure that prevented gate publication also prevents normal mutation-lock acquisition.
Correct the reported state-directory write failure, then run `$$nemoclaw <name> shields status` to resume recovery or receive exact-generation recovery guidance.
If the command reports exact-generation recovery guidance, stop all NemoClaw processes for the sandbox, then follow that guidance before you retry the snapshot.
If a timed shields-down window expires during snapshot work, the deadline gate blocks new mutations and waits for the exact snapshot owner to finish without signaling it.
Snapshot work does not bypass recovery for an expired shields-down window.
<AgentOnly variant="openclaw,hermes">
Follow [Timed Shields Windows](../configure-sandboxes/understand-runtime-changes#timed-shields-windows) to correct state-directory failures or complete exact-generation recovery before you rerun `$$nemoclaw <name> snapshot create`.
</AgentOnly>
<AgentOnly variant="deepagents">
If a state-directory failure prevents gate publication, normal mutation-lock acquisition also fails.
If recovery reports a state-directory failure, correct it and run `$$nemoclaw <name> shields status`.
Durable containment or retained exact gates keep new mutations blocked until you complete exact-generation recovery.
If the command reports exact-generation recovery guidance, stop all NemoClaw processes for the sandbox and follow that guidance before you rerun `$$nemoclaw <name> snapshot create`.
</AgentOnly>

Tag a snapshot with a human-readable label:

Expand Down
10 changes: 10 additions & 0 deletions docs/manage-sandboxes/runtime-controls.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -110,31 +110,41 @@ Run `$$nemoclaw <name> shields down` before the change, then restore lockdown wi
NemoClaw serializes host-side gateway recovery, config and inference writes, snapshot mutation, sandbox destruction, and shields transitions for each sandbox.
When `shields down --timeout` is active, each mutation binds to that exact timer generation so a replaced or expired timer cannot race a later command or a new sandbox that reuses the same name.

### Restore Lockdown After Expiration

If the timeout expires while a mutation is changing sandbox state, auto-restore closes the per-sandbox lifecycle deadline gate.
The gate blocks new mutations and waits for the recorded live owner to release its exact lock generation.
NemoClaw does not signal that process because portable process inspection cannot prove that all descendants stopped.
After the owner releases the lock, auto-restore restores the restrictive policy and configuration posture.
The ownership check includes both the process ID and process start identity so PID reuse does not grant control over an unrelated process.

### Complete Exact-Generation Recovery

An interactive command can take over an expired timer.
Interactive recovery has separate transition-takeover and restoration phases.
Each phase makes up to 7 attempts and waits 5 seconds between failures, for up to 30 seconds of retry delay per phase.
Detached recovery uses one 7-attempt budget across deadline setup, main-generation publication, and restoration.
The deadline gate remains closed during those attempts.

If restoration cannot commit, NemoClaw attempts to record durable containment.
If that containment commit also fails, NemoClaw retains any exact lifecycle and deadline gates it already owns.
A state-directory failure that prevented gate publication also prevents normal mutation-lock acquisition.
Correct the reported state-directory write failure, then run `$$nemoclaw <name> shields status` to resume recovery or receive exact-generation recovery guidance.
When recovery cannot complete, an interactive command returns an error, or the detached timer exits with a failure status.

NemoClaw also records durable containment when an owner exits before it can prove that the owner's descendants stopped, or when ownership becomes ambiguous.
Durable containment, retained exact gates, or the fail-closed state-directory error keeps new mutations blocked until you complete exact-generation operator recovery.

Stop all NemoClaw processes for the sandbox, then follow the paths, identities, tokens, and removal order in the reported error.
Verify each recorded generation is unchanged, remove only the exact stale generations first, and remove the exact containment generation last.

### Preserve Managed MCP Policy

Before a manual Shields transition replaces a policy, NemoClaw requires exact Model Context Protocol (MCP) agreement among the sandbox registry, generated-policy record, and live gateway policy.
`shields down` carries the proven managed MCP policy entries into the relaxed policy.
Restoration removes snapshot-time managed MCP entries before it overlays current exact entries.
If exact agreement is absent, a manual Shields transition refuses the replacement policy.

At an expired deadline, auto-restore omits unproven managed MCP policy entries, restores lockdown, and records the omission count in its audit entry.
An MCP server removed during the shields-down window stays removed.
A surviving server keeps its recorded endpoint and address pins while its policy ownership remains exact.
Expand Down
10 changes: 10 additions & 0 deletions docs/reference/architecture.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,16 @@ The router selects from the configured pool, then calls the upstream NVIDIA endp
Some model and provider combinations need agent-specific compatibility setup.
NemoClaw keeps those declarations under `nemoclaw-blueprint/model-specific-setup/<agent>/` so fixes for each supported agent can be tested and reviewed independently.

### Managed Inference Catalog

Managed inference profiles use definitions under `managed-inference/presets/` and `managed-inference/recipes/`.
The catalog compiler validates those definitions and includes them in the packaged runtime catalog.

For a managed-cluster recipe, the `nodeCount` field declares cluster cardinality.
Registered managed-cluster TypeScript adapters validate qualified topology and materialize each ranked node.
Contributors can define a compatible managed-cluster profile that uses an existing execution contract with preset and recipe YAML.
A managed-cluster profile that needs a different execution contract requires a registered adapter.

Refer to [Choose an Inference Provider](../inference/learn-and-choose/choose-inference-provider) for provider configuration details.

## Provider Credential Storage
Expand Down
Loading