fix(dcode): harden Nemotron Ultra tool requests - #6494
Conversation
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
…tra-profile-plugin Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
…tra-profile-plugin Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
…tra-profile-plugin Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
…tra-profile-plugin Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
…tra-profile-plugin Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: cjagwani <cjagwani@nvidia.com>
E2E Target Results —
|
| Job | Result |
|---|---|
| live |
E2E Target Results —
|
| Job | Result |
|---|---|
| live |
Signed-off-by: cjagwani <cjagwani@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
E2E Target Results — ❌ Some jobs failedRun: 28981919658
|
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
E2E Target Results — ✅ All selected jobs passedRun: 28983005052
|
Signed-off-by: cjagwani <cjagwani@nvidia.com>
E2E Target Results —
|
| Job | Result |
|---|---|
| live |
E2E Target Results — ✅ All selected jobs passedRun: 28983514326
|
<!-- markdownlint-disable MD041 --> ## Summary Prepares the user documentation for NemoClaw v0.0.78 by replacing the unreleased section with release highlights and synchronizing the affected inference, lifecycle, messaging, and CLI reference pages with merged behavior. ## Changes - Publish the v0.0.78 release-notes section with links to the most specific user guides for each shipped behavior. - Document authoritative Deep Agents route health, Nemotron Ultra profile behavior, and Hermes compatible-endpoint context metadata. - Document forced rebuild recovery after total backup failure and the ownership-safe tunnel/full-stop behavior. - Keep command examples and shared agent variants aligned with the current OpenClaw, Hermes, and Deep Agents interfaces. Source mapping: - [#3787](#3787) -> `docs/about/release-notes.mdx`: Record reliable workspace template seeding during sandbox startup. - [#4960](#4960) -> `docs/about/release-notes.mdx`: Record safer detection of rewritten OpenClaw gateway processes. - [#5676](#5676) -> `docs/about/release-notes.mdx`: Record warning-tolerant agent-list JSON handling. - [#5857](#5857) -> `docs/about/release-notes.mdx`: Record synchronization of explicit OpenClaw main-agent model state. - [#5929](#5929) -> `docs/about/release-notes.mdx`: Record copyable SSH port-forward guidance for remote dashboards. - [#6068](#6068) -> `docs/about/release-notes.mdx`: Record custom-image plugin provenance reconciliation. - [#6116](#6116) -> `docs/about/release-notes.mdx`: Record live-loopback dashboard-forward recovery. - [#6122](#6122) -> `docs/about/release-notes.mdx`: Announce validated, round-trippable policy YAML output. - [#6211](#6211) -> `docs/manage-sandboxes/lifecycle.mdx`, `docs/reference/commands.mdx`, `docs/about/release-notes.mdx`: Explain the explicit no-backup `rebuild --force` recovery boundary. - [#6283](#6283) -> `docs/about/release-notes.mdx`: Record Hermes WebUI port alignment. - [#6293](#6293) -> `docs/inference/switch-inference-providers.mdx`, `docs/about/release-notes.mdx`: Document compatible-endpoint context-length probing for Hermes. - [#6320](#6320) -> `docs/about/release-notes.mdx`: Record bounded gateway-recovery waits. - [#6377](#6377) -> `docs/reference/commands.mdx`, `docs/about/release-notes.mdx`: Explain rebuild diagnostics and prepared MCP-destroy recovery. - [#6412](#6412) -> `docs/get-started/quickstart-langchain-deepagents-code.mdx`, `docs/about/release-notes.mdx`: Document authoritative agent-visible inference route health. - [#6421](#6421) -> `docs/about/release-notes.mdx`: Record the longer quiet-pull window for managed vLLM images. - [#6431](#6431) -> `docs/inference/model-capability-audit.mdx`, `docs/about/release-notes.mdx`: Document the version-pinned Nemotron Ultra profile plugin. - [#6439](#6439) -> `docs/about/release-notes.mdx`: Summarize the authenticated, pinned credential-capture helper boundary. - [#6450](#6450) -> `docs/manage-sandboxes/messaging-channels.mdx`, `docs/reference/commands.mdx`, `docs/about/release-notes.mdx`: Document host-forward cleanup and ownership-safe gateway-port release. - [#6474](#6474) -> `docs/manage-sandboxes/messaging-channels.mdx`, `docs/about/release-notes.mdx`: Record composable OpenClaw messaging runtime loaders. - [#6475](#6475) -> `docs/about/release-notes.mdx`: Record removal of the unavailable Kimi K2.6 production endpoint option. - [#6480](#6480) -> `docs/about/release-notes.mdx`: Record stderr routing for the plugin registration banner. - [#6481](#6481) -> `docs/about/release-notes.mdx`: Record post-pull Ollama model discovery checks. - [#6482](#6482) -> `docs/about/release-notes.mdx`: Record Ollama model warm-up after daemon restart. - [#6486](#6486) -> `docs/about/release-notes.mdx`: Publish the opt-in, thread-scoped Deep Agents auto-approval boundary. - [#6490](#6490) -> `docs/about/release-notes.mdx`: Record diagnostics for custom images missing the managed runtime. - [#6494](#6494) -> `docs/inference/model-capability-audit.mdx`, `docs/about/release-notes.mdx`: Document nonempty tool-call content preservation and placeholder rejection. - [#6497](#6497) -> `docs/get-started/quickstart-langchain-deepagents-code.mdx`, `docs/about/release-notes.mdx`: Document isolated Deep Agents route-probe output. - [#6506](#6506) -> `docs/get-started/quickstart-langchain-deepagents-code.mdx`, `docs/about/release-notes.mdx`: Document observability-preserving managed route probes. - [#6508](#6508) -> `docs/about/release-notes.mdx`: Link the new extension taxonomy and SDK-readiness reference from the release summary. Release-source verification: GitHub reports all 29 cited source PRs as merged with base `main`, and every merge commit is an ancestor of `origin/main` at `17bf9a6a9688b3b1d69cf4b37d3f23110acb055e`. No source-mapping mismatches were found. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [x] Doc only (includes code sample changes) ## Quality Gates <!-- Check exactly one tests line and one docs line. Check other lines when applicable. Add every requested justification or approval reference. --> - [ ] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [x] Tests not applicable — justification: Documentation-only release-prep changes; `npm run docs` validates variants, routes, and Fern content. - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification <!-- Check each applicable item only when supported by the requested evidence. Run targeted tests once per relevant change set and rerun after later edits or hook autofixes that can affect the tested behavior. Do not rerun hook-covered checks. --> - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: Tests are not applicable to this documentation-only change set. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — exited 0 with zero errors; Fern reported the existing unauthenticated redirect-check and light-mode contrast warnings. - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Charan Jagwani <cjagwani@nvidia.com> --------- Signed-off-by: cjagwani <cjagwani@nvidia.com>
<!-- markdownlint-disable MD041 --> ## Summary Follow-up to #6431 for exact-head validation failures that completed immediately before that PR merged. This isolates sequential DCode E2E state, preserves observability across policy-only reloads, lets generic OpenClaw cloud-onboard runs skip DCode-only TUI prerequisites, and closes the remaining valid advisor requests without changing user-facing behavior. ## Related Issue Follow-up to #6431, which closed #6424. No new issue. ## Changes - Remove the durable Tavily preset after check 09, arm cleanup on `EXIT`, and fail unless managed DCode Python returns to the default network denial. - Move check 12's `expect`, Node, and timeout prerequisites after its DCode sandbox guard. Generic `cloud-onboard` therefore skips correctly; typed DCode jobs continue to require and install `expect`. - Keep the credential-free observability enable marker in managed `/sandbox/.deepagents` state for exec/login recovery. Current `main` now owns the durable lifecycle: absent entrypoint env preserves the bit across policy-only restarts, while create/rebuild/clone pass an explicit authoritative `1` or `0`. Check 09 requires the host registry's DCode `observabilityEnabled=true` intent and the exact persistent marker to agree before and after Tavily policy mutation; it fails rather than reconstructing intent from sandbox-writable state. Host-managed network policy remains the egress boundary. - Reject a pre-existing non-regular or symlink marker target with a clear startup error, and use no-target-directory replacement so a raced directory cannot absorb the enabled marker. Fixtures cover both enabled and disabled startup. - In live check 11, remove only `observability-otlp-local`, recreate the exact marker as the sandbox user, and prove both the normally allowed raw route and marker-triggered deterministic instrumentation produce zero host captures. Restore only from an exactly inactive policy state before the existing positive trace checks. - Replace repeated credential-name limit literals with `CREDENTIAL_NAME_PREFIX_MAX_LENGTH` and prove every Bash quantifier remains equal to the canonical TypeScript context-pattern limit. - Cover exact 128/129 context-prefix behavior, oversized OpenShell placeholders, and Tavily cleanup-command failures through the real wrapper/self-test boundaries. - Preserve the merged #6494 Nemotron Ultra request hardening and its observability create/snapshot lifecycle while resolving the overlapping follow-up checks against current `main`. - Keep standalone DCode Tavily persistence and rebuild behavior unchanged. - Avoid the rejected alternative of adding a privileged 47-line apt/workflow boundary to the generic cloud job. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: no CLI, configuration, model-ID, policy, or user-visible behavior changes - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — fresh exact-head review pending - [ ] Non-success, skipped, or missing CI check accepted by maintainer — none currently accepted ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — after merging current `main`, focused DCode integration suites pass 210/210 and focused E2E-support suites pass 37/37. Bash syntax, ShellCheck, source-shape, test-size, build, TypeScript typecheck, commit, and pre-push hooks pass - [ ] Applicable broad gate passed — fresh exact-head CI, advisors, focused DCode E2E, cloud-onboard, and credential-sanitization pending - [ ] Quality Gates section completed with required justifications or waivers — exact-head sensitive-path review pending - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) ## Advisor dispositions - Source-of-truth review: the invalid state is a managed DCode exec losing host-selected observability because raw exec/login processes do not inherit entrypoint environment and policy mutation reloads do not re-run the entrypoint. Current `main` now owns `start.sh` materialization plus `dcode-launcher.sh` recovery under `/sandbox/.deepagents`; create/rebuild/clone carry the registry's explicit `1`/`0` intent, and snapshot restore honors that authoritative registry state so an earlier enabled snapshot does not override a later opt-out. This PR adds fail-closed registry/marker agreement checks and the no-policy export proof. Remove the bridge only when OpenShell both propagates the bit to exec/login processes and preserves it through policy reloads or re-runs the entrypoint. - Sequential Tavily isolation is exercised by the required typed target on one sandbox: check 09 proves denial immediately after cleanup; checks 10 and 11 do not mutate Tavily; check 12 rebuilds and then invokes check 06, which proves both managed and project-venv Python remain denied from `api.tavily.com`. This gives immediate and post-rebuild denial evidence without redundant probes after non-mutating checks. - The marker integration test runs the transformed `start.sh` with `NEMOCLAW_OBSERVABILITY=1`, verifies the `/sandbox/.deepagents/.nemoclaw-observability-enabled` value and mode, removes volatile runtime state, proves absent-env restart and launcher recovery, and proves explicit `0` removal. Live check 11 verifies the literal sandbox path. - The `/sandbox/.deepagents` marker is intentionally untrusted convenience state, not authorization. A sandbox user can request local instrumentation by recreating the exact marker, but cannot grant OTLP egress; the fixed exporter route remains subject to host-managed `observability-otlp-local` policy. Live check 11 removes that host policy, recreates the marker as the sandbox user, proves the exact raw route is denied without capture, and runs real marker-triggered deterministic instrumentation with zero captures before restoring the policy. It also retains alternate host/path/method/port and unmanaged-binary denials with policy active. - Check 09 no longer repairs observability by re-running the stateful entrypoint. The host registry is authoritative, the persistent marker is derived convenience state, and disagreement fails closed both before and after policy mutation. Merged #6506 prevents status/connect route probes from clearing that marker by using the side-effect-free managed-exec boundary. - Startup rejects directory, symlink, and other non-regular marker targets before either the enabled or disabled branch. Enabled replacement treats the destination as a file, closing the directory-descent edge case raised by the exact-head advisor. - Current `main` includes #6506's side-effect-free managed-exec boundary and #6494's durable observability lifecycle. Both are retained together with this PR's host-registry agreement checks and no-policy proof; no contributor work was replaced. - The proxy env file remains intentionally volatile in `/tmp`: each stateful entrypoint reconstructs it from root-owned image proxy inputs, while the observability marker under `/sandbox/.deepagents` must survive policy-only reloads that do not carry the sandbox-create environment. Their different locations reflect different lifetime contracts, not inconsistent authority. ## Exact-head validation plan - Re-run automatic CI, CodeRabbit, GPT, and Nemotron advisors on `0570b876d6e85bcb110823d5f8b6a5553d266f34`. - Re-run `ubuntu-repo-cloud-langchain-deepagents-code` to prove Tavily is denied after check 09 cleanup and remains denied across the check 12 rebuild. - Re-run `cloud-onboard` to prove the non-DCode sandbox skips before requiring `expect`. - Re-run `credential-sanitization` for the named-limit parity change. - Hold merge until tomorrow even if all gates pass earlier. ## Failure evidence addressed - Focused DCode run [28978277604](https://github.com/NVIDIA/NemoClaw/actions/runs/28978277604): check 09 left durable Tavily state that check 12 correctly replayed before the strict egress boundary. - Cloud/credential run [28978281322](https://github.com/NVIDIA/NemoClaw/actions/runs/28978281322): generic OpenClaw cloud-onboard required `expect` before reaching its DCode-only skip; credential-sanitization passed. - Superseded follow-up run [28980032707](https://github.com/NVIDIA/NemoClaw/actions/runs/28980032707): Tavily cleanup and strict egress passed, then check 11 proved that policy reload had cleared the runtime-functional `/tmp` observability marker. The current head retains current `main`'s durable marker under `/sandbox/.deepagents`. - Superseded follow-up run [28980032711](https://github.com/NVIDIA/NemoClaw/actions/runs/28980032711): cloud-onboard and credential-sanitization both passed. - Superseded follow-up run [28981014655](https://github.com/NVIDIA/NemoClaw/actions/runs/28981014655): after the marker moved to `/sandbox`, check 11 proved that Tavily policy cleanup could still reconstruct runtime convenience state and remove it. The current head uses current `main`'s absent-env-preserves lifecycle and fails if the authoritative registry and derived marker disagree; check 09 does not repair drift. - Superseded follow-up run [28981369440](https://github.com/NVIDIA/NemoClaw/actions/runs/28981369440): cloud-onboard and credential-sanitization both passed on the collaborator head; exact-head rerun pending. - Superseded follow-up run [28981690759](https://github.com/NVIDIA/NemoClaw/actions/runs/28981690759): check 09 restored Tavily but sampled observability after policy-add had already reloaded the sandbox, so check 11 found the marker absent. This showed that sandbox marker sampling cannot be the authoritative source of host intent. - Superseded follow-up run [28982642290](https://github.com/NVIDIA/NemoClaw/actions/runs/28982642290): check 09 restored Tavily and check 10 passed, but check 11 found the marker absent because a pre-#6506 status/connect route probe had already invoked the stateful entrypoint without observability. Current `main` supplies #6506's side-effect-free managed-exec probe and #6494's explicit observability lifecycle, while this head compares the host registry and persistent marker before and after policy mutation instead of masking drift with a repair. - Superseded exact-head focused run [28983221993](https://github.com/NVIDIA/NemoClaw/actions/runs/28983221993) passed every typed DCode check 03–12: check 09 reported 9/9, check 11 reported 11/11, and check 12 proved post-rebuild denial. Combined run [28983222022](https://github.com/NVIDIA/NemoClaw/actions/runs/28983222022) passed cloud-onboard and credential sanitization. These runs were invalidated only to address the advisor's unsafe-target edge case and add the stricter no-host-policy leak proof now running on the current head. - Superseded advisor run [28983777162](https://github.com/NVIDIA/NemoClaw/actions/runs/28983777162) rated the safety changes `merge_as_is` from GPT with no findings; Nemotron agreed both substantive findings were resolved and requested only the `/tmp` versus `/sandbox` lifetime cross-reference now present on the current head. --- Signed-off-by: Aaron Erickson <aerickson@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved validation for credential names and context placeholders, including oversized names. * Strengthened proxy environment file handling to prevent destination-path issues. * Clarified and stabilized observability policy and marker handling. * **Tests** * Expanded end-to-end coverage for observability, Tavily access, tracing, and auto-approval behavior. * Standardized test environment paths and added boundary-case validation for credential patterns. * Improved platform-specific launcher test reliability. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Aaron Erickson <aerickson@nvidia.com> Signed-off-by: cjagwani <cjagwani@nvidia.com> Co-authored-by: cjagwani <cjagwani@nvidia.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com>
## Summary Harden the managed Nemotron 3 Ultra DCode aliases at two separate compatibility boundaries: emit the model-specific nonempty-content template argument required for reasoning-plus-tool-call requests, and reject the observed literal `[content]` execute placeholder before it reaches shell dispatch. NVIDIA#6431 is merged; this PR is based directly on current `main`. It leaves the canonical NVIDIA profile and all unrelated models unchanged. ## Source boundary and lifecycle - `force_nonempty_content`: the invalid empty-assistant-content state originates in the Ultra serving template. NemoClaw owns only the generated per-model request configuration. Remove the override after a reviewed serving/client update works for both managed Ultra IDs without it and the live DCode E2E passes. - Execute guard: the invalid state is a model-produced tool call whose complete `execute.command` is the literal `[content]` (ignoring ASCII whitespace around the token and brackets, and case). NemoClaw owns the managed alias middleware immediately before dispatch, not the model, serving template, or hash-locked canonical Deep Agents profile. Remove the guard only after those upstream paths no longer produce or convert the placeholder and the focused plus live tests pass with it deleted. ## Changes - Generate `force_nonempty_content = true` only for the two managed Ultra model IDs. - Add a managed-alias-only sync/async middleware guard for the exact placeholder; concrete execute commands and other tools pass through unchanged. - Validate rejection through the pinned LangChain `ToolMessage.content` API in the image validator, fixture, and hosted E2E probe. - Preserve atomic/idempotent profile registration with the process-local registry as source of truth and a lock around multi-key registration. - Persist the credential-free observability enable bit across environment-less policy restarts; create/rebuild/clone paths pass an explicit `1` or `0`. - Record invalid-state sources, regression evidence, and removal conditions in the dependency review. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: no CLI, setting, supported-model choice, or operator workflow changes; the internal dependency review is updated - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: pending maintainer review - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — combined Ultra, lifecycle, current-main, and acceptance suites passed 159/159; the post-hook launcher/acceptance subset passed 33/33 - [ ] Applicable broad gate passed — final exact-head regular CI and hosted Deep Agents Code E2E are running - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: Aaron Erickson <aerickson@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added model-specific “Ultra” configuration for selected OpenAI-compatible model IDs to enforce non-empty tool-call/reasoning content. * Added managed execute placeholder guard behavior and deeper end-to-end validation for Tavily policy remove/restore ordering and observability recovery. * **Bug Fixes** * Hardened managed execute handling to reject placeholder `execute` calls (including async/sync parity) while allowing concrete commands. * Improved managed alias registration for safer atomic idempotence and rollback. * **Documentation** * Updated compatibility/validation guidance and the dependency advisory review hash. * **Tests** * Expanded E2E/profile/config contract tests, including workflow host-dependency step ordering and guard dispatch assertions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Aaron Erickson <aerickson@nvidia.com> Signed-off-by: Carlos Villela <cvillela@nvidia.com> Signed-off-by: cjagwani <cjagwani@nvidia.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com> Co-authored-by: cjagwani <cjagwani@nvidia.com>
<!-- markdownlint-disable MD041 --> ## Summary Prepares the user documentation for NemoClaw v0.0.78 by replacing the unreleased section with release highlights and synchronizing the affected inference, lifecycle, messaging, and CLI reference pages with merged behavior. ## Changes - Publish the v0.0.78 release-notes section with links to the most specific user guides for each shipped behavior. - Document authoritative Deep Agents route health, Nemotron Ultra profile behavior, and Hermes compatible-endpoint context metadata. - Document forced rebuild recovery after total backup failure and the ownership-safe tunnel/full-stop behavior. - Keep command examples and shared agent variants aligned with the current OpenClaw, Hermes, and Deep Agents interfaces. Source mapping: - [NVIDIA#3787](NVIDIA#3787) -> `docs/about/release-notes.mdx`: Record reliable workspace template seeding during sandbox startup. - [NVIDIA#4960](NVIDIA#4960) -> `docs/about/release-notes.mdx`: Record safer detection of rewritten OpenClaw gateway processes. - [NVIDIA#5676](NVIDIA#5676) -> `docs/about/release-notes.mdx`: Record warning-tolerant agent-list JSON handling. - [NVIDIA#5857](NVIDIA#5857) -> `docs/about/release-notes.mdx`: Record synchronization of explicit OpenClaw main-agent model state. - [NVIDIA#5929](NVIDIA#5929) -> `docs/about/release-notes.mdx`: Record copyable SSH port-forward guidance for remote dashboards. - [NVIDIA#6068](NVIDIA#6068) -> `docs/about/release-notes.mdx`: Record custom-image plugin provenance reconciliation. - [NVIDIA#6116](NVIDIA#6116) -> `docs/about/release-notes.mdx`: Record live-loopback dashboard-forward recovery. - [NVIDIA#6122](NVIDIA#6122) -> `docs/about/release-notes.mdx`: Announce validated, round-trippable policy YAML output. - [NVIDIA#6211](NVIDIA#6211) -> `docs/manage-sandboxes/lifecycle.mdx`, `docs/reference/commands.mdx`, `docs/about/release-notes.mdx`: Explain the explicit no-backup `rebuild --force` recovery boundary. - [NVIDIA#6283](NVIDIA#6283) -> `docs/about/release-notes.mdx`: Record Hermes WebUI port alignment. - [NVIDIA#6293](NVIDIA#6293) -> `docs/inference/switch-inference-providers.mdx`, `docs/about/release-notes.mdx`: Document compatible-endpoint context-length probing for Hermes. - [NVIDIA#6320](NVIDIA#6320) -> `docs/about/release-notes.mdx`: Record bounded gateway-recovery waits. - [NVIDIA#6377](NVIDIA#6377) -> `docs/reference/commands.mdx`, `docs/about/release-notes.mdx`: Explain rebuild diagnostics and prepared MCP-destroy recovery. - [NVIDIA#6412](NVIDIA#6412) -> `docs/get-started/quickstart-langchain-deepagents-code.mdx`, `docs/about/release-notes.mdx`: Document authoritative agent-visible inference route health. - [NVIDIA#6421](NVIDIA#6421) -> `docs/about/release-notes.mdx`: Record the longer quiet-pull window for managed vLLM images. - [NVIDIA#6431](NVIDIA#6431) -> `docs/inference/model-capability-audit.mdx`, `docs/about/release-notes.mdx`: Document the version-pinned Nemotron Ultra profile plugin. - [NVIDIA#6439](NVIDIA#6439) -> `docs/about/release-notes.mdx`: Summarize the authenticated, pinned credential-capture helper boundary. - [NVIDIA#6450](NVIDIA#6450) -> `docs/manage-sandboxes/messaging-channels.mdx`, `docs/reference/commands.mdx`, `docs/about/release-notes.mdx`: Document host-forward cleanup and ownership-safe gateway-port release. - [NVIDIA#6474](NVIDIA#6474) -> `docs/manage-sandboxes/messaging-channels.mdx`, `docs/about/release-notes.mdx`: Record composable OpenClaw messaging runtime loaders. - [NVIDIA#6475](NVIDIA#6475) -> `docs/about/release-notes.mdx`: Record removal of the unavailable Kimi K2.6 production endpoint option. - [NVIDIA#6480](NVIDIA#6480) -> `docs/about/release-notes.mdx`: Record stderr routing for the plugin registration banner. - [NVIDIA#6481](NVIDIA#6481) -> `docs/about/release-notes.mdx`: Record post-pull Ollama model discovery checks. - [NVIDIA#6482](NVIDIA#6482) -> `docs/about/release-notes.mdx`: Record Ollama model warm-up after daemon restart. - [NVIDIA#6486](NVIDIA#6486) -> `docs/about/release-notes.mdx`: Publish the opt-in, thread-scoped Deep Agents auto-approval boundary. - [NVIDIA#6490](NVIDIA#6490) -> `docs/about/release-notes.mdx`: Record diagnostics for custom images missing the managed runtime. - [NVIDIA#6494](NVIDIA#6494) -> `docs/inference/model-capability-audit.mdx`, `docs/about/release-notes.mdx`: Document nonempty tool-call content preservation and placeholder rejection. - [NVIDIA#6497](NVIDIA#6497) -> `docs/get-started/quickstart-langchain-deepagents-code.mdx`, `docs/about/release-notes.mdx`: Document isolated Deep Agents route-probe output. - [NVIDIA#6506](NVIDIA#6506) -> `docs/get-started/quickstart-langchain-deepagents-code.mdx`, `docs/about/release-notes.mdx`: Document observability-preserving managed route probes. - [NVIDIA#6508](NVIDIA#6508) -> `docs/about/release-notes.mdx`: Link the new extension taxonomy and SDK-readiness reference from the release summary. Release-source verification: GitHub reports all 29 cited source PRs as merged with base `main`, and every merge commit is an ancestor of `origin/main` at `17bf9a6a9688b3b1d69cf4b37d3f23110acb055e`. No source-mapping mismatches were found. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [x] Doc only (includes code sample changes) ## Quality Gates <!-- Check exactly one tests line and one docs line. Check other lines when applicable. Add every requested justification or approval reference. --> - [ ] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [x] Tests not applicable — justification: Documentation-only release-prep changes; `npm run docs` validates variants, routes, and Fern content. - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification <!-- Check each applicable item only when supported by the requested evidence. Run targeted tests once per relevant change set and rerun after later edits or hook autofixes that can affect the tested behavior. Do not rerun hook-covered checks. --> - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: Tests are not applicable to this documentation-only change set. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — exited 0 with zero errors; Fern reported the existing unauthenticated redirect-check and light-mode contrast warnings. - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Charan Jagwani <cjagwani@nvidia.com> --------- Signed-off-by: cjagwani <cjagwani@nvidia.com>
<!-- markdownlint-disable MD041 --> ## Summary Follow-up to NVIDIA#6431 for exact-head validation failures that completed immediately before that PR merged. This isolates sequential DCode E2E state, preserves observability across policy-only reloads, lets generic OpenClaw cloud-onboard runs skip DCode-only TUI prerequisites, and closes the remaining valid advisor requests without changing user-facing behavior. ## Related Issue Follow-up to NVIDIA#6431, which closed NVIDIA#6424. No new issue. ## Changes - Remove the durable Tavily preset after check 09, arm cleanup on `EXIT`, and fail unless managed DCode Python returns to the default network denial. - Move check 12's `expect`, Node, and timeout prerequisites after its DCode sandbox guard. Generic `cloud-onboard` therefore skips correctly; typed DCode jobs continue to require and install `expect`. - Keep the credential-free observability enable marker in managed `/sandbox/.deepagents` state for exec/login recovery. Current `main` now owns the durable lifecycle: absent entrypoint env preserves the bit across policy-only restarts, while create/rebuild/clone pass an explicit authoritative `1` or `0`. Check 09 requires the host registry's DCode `observabilityEnabled=true` intent and the exact persistent marker to agree before and after Tavily policy mutation; it fails rather than reconstructing intent from sandbox-writable state. Host-managed network policy remains the egress boundary. - Reject a pre-existing non-regular or symlink marker target with a clear startup error, and use no-target-directory replacement so a raced directory cannot absorb the enabled marker. Fixtures cover both enabled and disabled startup. - In live check 11, remove only `observability-otlp-local`, recreate the exact marker as the sandbox user, and prove both the normally allowed raw route and marker-triggered deterministic instrumentation produce zero host captures. Restore only from an exactly inactive policy state before the existing positive trace checks. - Replace repeated credential-name limit literals with `CREDENTIAL_NAME_PREFIX_MAX_LENGTH` and prove every Bash quantifier remains equal to the canonical TypeScript context-pattern limit. - Cover exact 128/129 context-prefix behavior, oversized OpenShell placeholders, and Tavily cleanup-command failures through the real wrapper/self-test boundaries. - Preserve the merged NVIDIA#6494 Nemotron Ultra request hardening and its observability create/snapshot lifecycle while resolving the overlapping follow-up checks against current `main`. - Keep standalone DCode Tavily persistence and rebuild behavior unchanged. - Avoid the rejected alternative of adding a privileged 47-line apt/workflow boundary to the generic cloud job. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: no CLI, configuration, model-ID, policy, or user-visible behavior changes - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — fresh exact-head review pending - [ ] Non-success, skipped, or missing CI check accepted by maintainer — none currently accepted ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — after merging current `main`, focused DCode integration suites pass 210/210 and focused E2E-support suites pass 37/37. Bash syntax, ShellCheck, source-shape, test-size, build, TypeScript typecheck, commit, and pre-push hooks pass - [ ] Applicable broad gate passed — fresh exact-head CI, advisors, focused DCode E2E, cloud-onboard, and credential-sanitization pending - [ ] Quality Gates section completed with required justifications or waivers — exact-head sensitive-path review pending - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) ## Advisor dispositions - Source-of-truth review: the invalid state is a managed DCode exec losing host-selected observability because raw exec/login processes do not inherit entrypoint environment and policy mutation reloads do not re-run the entrypoint. Current `main` now owns `start.sh` materialization plus `dcode-launcher.sh` recovery under `/sandbox/.deepagents`; create/rebuild/clone carry the registry's explicit `1`/`0` intent, and snapshot restore honors that authoritative registry state so an earlier enabled snapshot does not override a later opt-out. This PR adds fail-closed registry/marker agreement checks and the no-policy export proof. Remove the bridge only when OpenShell both propagates the bit to exec/login processes and preserves it through policy reloads or re-runs the entrypoint. - Sequential Tavily isolation is exercised by the required typed target on one sandbox: check 09 proves denial immediately after cleanup; checks 10 and 11 do not mutate Tavily; check 12 rebuilds and then invokes check 06, which proves both managed and project-venv Python remain denied from `api.tavily.com`. This gives immediate and post-rebuild denial evidence without redundant probes after non-mutating checks. - The marker integration test runs the transformed `start.sh` with `NEMOCLAW_OBSERVABILITY=1`, verifies the `/sandbox/.deepagents/.nemoclaw-observability-enabled` value and mode, removes volatile runtime state, proves absent-env restart and launcher recovery, and proves explicit `0` removal. Live check 11 verifies the literal sandbox path. - The `/sandbox/.deepagents` marker is intentionally untrusted convenience state, not authorization. A sandbox user can request local instrumentation by recreating the exact marker, but cannot grant OTLP egress; the fixed exporter route remains subject to host-managed `observability-otlp-local` policy. Live check 11 removes that host policy, recreates the marker as the sandbox user, proves the exact raw route is denied without capture, and runs real marker-triggered deterministic instrumentation with zero captures before restoring the policy. It also retains alternate host/path/method/port and unmanaged-binary denials with policy active. - Check 09 no longer repairs observability by re-running the stateful entrypoint. The host registry is authoritative, the persistent marker is derived convenience state, and disagreement fails closed both before and after policy mutation. Merged NVIDIA#6506 prevents status/connect route probes from clearing that marker by using the side-effect-free managed-exec boundary. - Startup rejects directory, symlink, and other non-regular marker targets before either the enabled or disabled branch. Enabled replacement treats the destination as a file, closing the directory-descent edge case raised by the exact-head advisor. - Current `main` includes NVIDIA#6506's side-effect-free managed-exec boundary and NVIDIA#6494's durable observability lifecycle. Both are retained together with this PR's host-registry agreement checks and no-policy proof; no contributor work was replaced. - The proxy env file remains intentionally volatile in `/tmp`: each stateful entrypoint reconstructs it from root-owned image proxy inputs, while the observability marker under `/sandbox/.deepagents` must survive policy-only reloads that do not carry the sandbox-create environment. Their different locations reflect different lifetime contracts, not inconsistent authority. ## Exact-head validation plan - Re-run automatic CI, CodeRabbit, GPT, and Nemotron advisors on `0570b876d6e85bcb110823d5f8b6a5553d266f34`. - Re-run `ubuntu-repo-cloud-langchain-deepagents-code` to prove Tavily is denied after check 09 cleanup and remains denied across the check 12 rebuild. - Re-run `cloud-onboard` to prove the non-DCode sandbox skips before requiring `expect`. - Re-run `credential-sanitization` for the named-limit parity change. - Hold merge until tomorrow even if all gates pass earlier. ## Failure evidence addressed - Focused DCode run [28978277604](https://github.com/NVIDIA/NemoClaw/actions/runs/28978277604): check 09 left durable Tavily state that check 12 correctly replayed before the strict egress boundary. - Cloud/credential run [28978281322](https://github.com/NVIDIA/NemoClaw/actions/runs/28978281322): generic OpenClaw cloud-onboard required `expect` before reaching its DCode-only skip; credential-sanitization passed. - Superseded follow-up run [28980032707](https://github.com/NVIDIA/NemoClaw/actions/runs/28980032707): Tavily cleanup and strict egress passed, then check 11 proved that policy reload had cleared the runtime-functional `/tmp` observability marker. The current head retains current `main`'s durable marker under `/sandbox/.deepagents`. - Superseded follow-up run [28980032711](https://github.com/NVIDIA/NemoClaw/actions/runs/28980032711): cloud-onboard and credential-sanitization both passed. - Superseded follow-up run [28981014655](https://github.com/NVIDIA/NemoClaw/actions/runs/28981014655): after the marker moved to `/sandbox`, check 11 proved that Tavily policy cleanup could still reconstruct runtime convenience state and remove it. The current head uses current `main`'s absent-env-preserves lifecycle and fails if the authoritative registry and derived marker disagree; check 09 does not repair drift. - Superseded follow-up run [28981369440](https://github.com/NVIDIA/NemoClaw/actions/runs/28981369440): cloud-onboard and credential-sanitization both passed on the collaborator head; exact-head rerun pending. - Superseded follow-up run [28981690759](https://github.com/NVIDIA/NemoClaw/actions/runs/28981690759): check 09 restored Tavily but sampled observability after policy-add had already reloaded the sandbox, so check 11 found the marker absent. This showed that sandbox marker sampling cannot be the authoritative source of host intent. - Superseded follow-up run [28982642290](https://github.com/NVIDIA/NemoClaw/actions/runs/28982642290): check 09 restored Tavily and check 10 passed, but check 11 found the marker absent because a pre-NVIDIA#6506 status/connect route probe had already invoked the stateful entrypoint without observability. Current `main` supplies NVIDIA#6506's side-effect-free managed-exec probe and NVIDIA#6494's explicit observability lifecycle, while this head compares the host registry and persistent marker before and after policy mutation instead of masking drift with a repair. - Superseded exact-head focused run [28983221993](https://github.com/NVIDIA/NemoClaw/actions/runs/28983221993) passed every typed DCode check 03–12: check 09 reported 9/9, check 11 reported 11/11, and check 12 proved post-rebuild denial. Combined run [28983222022](https://github.com/NVIDIA/NemoClaw/actions/runs/28983222022) passed cloud-onboard and credential sanitization. These runs were invalidated only to address the advisor's unsafe-target edge case and add the stricter no-host-policy leak proof now running on the current head. - Superseded advisor run [28983777162](https://github.com/NVIDIA/NemoClaw/actions/runs/28983777162) rated the safety changes `merge_as_is` from GPT with no findings; Nemotron agreed both substantive findings were resolved and requested only the `/tmp` versus `/sandbox` lifetime cross-reference now present on the current head. --- Signed-off-by: Aaron Erickson <aerickson@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved validation for credential names and context placeholders, including oversized names. * Strengthened proxy environment file handling to prevent destination-path issues. * Clarified and stabilized observability policy and marker handling. * **Tests** * Expanded end-to-end coverage for observability, Tavily access, tracing, and auto-approval behavior. * Standardized test environment paths and added boundary-case validation for credential patterns. * Improved platform-specific launcher test reliability. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Aaron Erickson <aerickson@nvidia.com> Signed-off-by: cjagwani <cjagwani@nvidia.com> Co-authored-by: cjagwani <cjagwani@nvidia.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com>
Summary
Harden the managed Nemotron 3 Ultra DCode aliases at two separate compatibility
boundaries: emit the model-specific nonempty-content template argument required
for reasoning-plus-tool-call requests, and reject the observed literal
[content]execute placeholder before it reaches shell dispatch.#6431 is merged; this PR is based directly on current
main. It leaves thecanonical NVIDIA profile and all unrelated models unchanged.
Source boundary and lifecycle
force_nonempty_content: the invalid empty-assistant-content state originatesin the Ultra serving template. NemoClaw owns only the generated per-model
request configuration. Remove the override after a reviewed serving/client
update works for both managed Ultra IDs without it and the live DCode E2E passes.
execute.commandis the literal[content](ignoring ASCII whitespace around the token andbrackets, and case). NemoClaw owns the managed alias middleware immediately before dispatch,
not the model, serving template, or hash-locked canonical Deep Agents profile.
Remove the guard only after those upstream paths no longer produce or convert
the placeholder and the focused plus live tests pass with it deleted.
Changes
force_nonempty_content = trueonly for the two managed Ultra model IDs.concrete execute commands and other tools pass through unchanged.
ToolMessage.contentAPI inthe image validator, fixture, and hosted E2E probe.
registry as source of truth and a lock around multi-key registration.
policy restarts; create/rebuild/clone paths pass an explicit
1or0.the dependency review.
Type of Change
Quality Gates
Verification
Verifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailablenpm run docsbuilds without warnings (doc changes only)Signed-off-by: Aaron Erickson aerickson@nvidia.com
Summary by CodeRabbit
executecalls (including async/sync parity) while allowing concrete commands.