Skip to content

fix(sandbox/recover): revert Hermes probe boundary enforcement - #5547

Closed
sandl99 wants to merge 4 commits into
mainfrom
revert-5530-fix/hermes-secret-boundary-on-recover-probe
Closed

fix(sandbox/recover): revert Hermes probe boundary enforcement#5547
sandl99 wants to merge 4 commits into
mainfrom
revert-5530-fix/hermes-secret-boundary-on-recover-probe

Conversation

@sandl99

@sandl99 sandl99 commented Jun 19, 2026

Copy link
Copy Markdown
Collaborator

#5530 break

NemoClaw git:(main) ✗ nemohermes sb connect      

  ⚠ Sandbox 'sb' is running Hermes Agent 0.14.0 (current: 2026.5.16)
    Run: nemoclaw sb rebuild


  Secret-boundary check could not run against the Hermes gateway in 'sb'.
  Refusing recovery to keep the validator-enforced boundary intact.

  Connect failed: secret-boundary check did not complete for Hermes Agent gateway in 'sb'.
  Inspect the validator output above and re-run `nemoclaw <sandbox> recover`.

Signed-off-by: San Dang sdang@nvidia.com

Summary by CodeRabbit

  • Bug Fixes

    • Removed secret boundary validation from sandbox recovery and connection operations.
  • Documentation

    • Updated command reference documentation to remove Hermes-specific recovery validation instructions.

@sandl99
sandl99 requested a review from laitingsheng June 19, 2026 09:21
@coderabbitai

coderabbitai Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Removes the Hermes secret-boundary enforcement feature end-to-end: deletes exported marker constants and the buildHermesEnvFileBoundaryStandaloneCheck() function from hermes-recovery-boundary.ts, strips the boundary-gating branch from checkAndRecoverSandboxProcesses, removes secretBoundaryRefused handling from connect/probe flows, simplifies test mocks, and removes the corresponding documentation paragraphs.

Changes

Hermes Secret-Boundary Enforcement Removal

Layer / File(s) Summary
Boundary module: remove exported constants and standalone check
src/lib/agent/hermes-recovery-boundary.ts, src/lib/agent/runtime-hermes-secret-boundary-behavioural.test.ts
Removes the three exported marker constants (SECRET_BOUNDARY_REFUSED_MARKER, SECRET_BOUNDARY_OK_MARKER, SECRET_BOUNDARY_VALIDATOR_MISSING_MARKER) and the standalone buildHermesEnvFileBoundaryStandaloneCheck() function. Updates the __testing export to expose only the two remaining guard builders. Removes corresponding standalone env-file check test cases from the behavioural test file.
Process recovery: remove secret-boundary gating and test cases
src/lib/actions/sandbox/process-recovery.ts, test/process-recovery.test.ts
Removes the boundary-validator import, the enforcement helper, and the early-return refusal branch from checkAndRecoverSandboxProcesses. The alive-gateway path now proceeds directly to forward/process health checks without secret-boundary gating. Five existing spawnSync mocks are simplified to unconditional mockReturnValue, and ~528 lines of secret-boundary test cases are deleted.
Connect flow: remove secretBoundaryRefused handling
src/lib/actions/sandbox/connect.ts, src/lib/actions/sandbox/connect-flow.test.ts
Relocates the getSandboxTargetGatewayName import, deletes the exitOnSecretBoundaryRefusal helper, removes the secretBoundaryRefused check from runSandboxConnectProbe and the result assignment/branch from connectSandbox. Removes secretBoundaryRefused/secretBoundaryReason from the test harness processCheck type and deletes boundary-refusal test cases.
Documentation: remove secret-boundary recover paragraphs
docs/reference/commands-nemohermes.mdx, docs/reference/commands.mdx
Removes the Hermes-specific paragraphs from the recover command sections in both reference docs, covering per-run secret-boundary re-evaluation, raw-secret rejection, and the older-image [boundary] warning/skip behavior.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#5530: Directly modifies the same hermes-recovery-boundary.ts, process-recovery.ts, and connect.ts files — adds the probe/standalone boundary checks that this PR removes.
  • NVIDIA/NemoClaw#5342: Touches runtime-hermes-secret-boundary-behavioural.test.ts and SECRET_BOUNDARY_REFUSED handling in the same recovery boundary path that this PR deletes.
  • NVIDIA/NemoClaw#5389: Modifies checkAndRecoverSandboxProcesses in process-recovery.ts for declared forward_ports re-establishment, directly intersecting the same recovery function modified here.

Suggested labels

bug-fix, area: sandbox, integration: hermes, area: docs, v0.0.66

Suggested reviewers

  • cv

Poem

🐇 Hop, hop, away with the boundary wall!
No more secret-shaped values to stall,
The gateway runs free with no refusal in sight,
Five mocks made simpler, the tests now feel light.
Docs trimmed and tidy, the hermes path clear —
Less code to maintain, more carrots this year! 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main change: reverting Hermes probe boundary enforcement in sandbox recovery. It directly matches the PR's primary objective of reverting PR #5530 that broke nemohermes functionality.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch revert-5530-fix/hermes-secret-boundary-on-recover-probe

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-code-quality

github-code-quality Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the revert-5530-fix/herm... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main revert-5530-fix/herm... 4b49214 +/-
nemoclaw/src/se...cret-scanner.ts 100%
nemoclaw/src/commands/slash.ts 100%
nemoclaw/src/li...bprocess-env.ts 100%
nemoclaw/src/bl...eprint/state.ts 98%
nemoclaw/src/onboard/config.ts 98%
nemoclaw/src/bl...int/snapshot.ts 97%
nemoclaw/src/bl...print/runner.ts 95%
nemoclaw/src/co...ration-state.ts 94%
nemoclaw/src/bl...ate-networks.ts 94%
nemoclaw/src/index.ts 94%

TypeScript / code-coverage/cli

The overall coverage in the revert-5530-fix/herm... branch is 46%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main revert-5530-fix/herm... 4b49214 +/-
src/lib/state/o...oard-session.ts 91%
src/lib/inference/local.ts 76%
src/lib/sandbox/config.ts 72%
src/lib/actions...dbox/rebuild.ts 67%
src/lib/onboard/preflight.ts 64%
src/lib/actions...licy-channel.ts 56%
src/lib/state/sandbox.ts 55%
src/lib/policy/index.ts 49%
src/lib/onboard...er-gpu-patch.ts 44%
src/lib/onboard.ts 18%

Updated June 22, 2026 16:18 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Blocked

Merge posture: Do not merge until addressed
Primary next action: Fix PRA-2: Restore Hermes running-gateway secret-boundary enforcement; then add or justify PRA-T1.
Open items: 3 required · 1 warning · 0 suggestions · 8 test follow-ups
Since last review: 0 prior items resolved · 4 still apply · 0 new items found

Action checklist

  • PRA-2 Fix: Restore Hermes running-gateway secret-boundary enforcement in src/lib/actions/sandbox/process-recovery.ts:500
  • PRA-3 Fix: Add a reproducer or narrow replacement for the claimed connect/recover break in test/process-recovery.test.ts:245
  • PRA-4 Fix: Restore negative coverage that stops connect side effects after boundary refusal in src/lib/actions/sandbox/connect-flow.test.ts:146
  • PRA-1 Resolve or justify: Source-of-truth review needed: Hermes recover/connect workaround reverting fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530
  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Runtime validation
  • PRA-T4 Add or justify test follow-up: Runtime validation
  • PRA-T5 Add or justify test follow-up: Runtime validation
  • PRA-T6 Add or justify test follow-up: Restore negative coverage that stops connect side effects after boundary refusal
  • PRA-T7 Add or justify test follow-up: Acceptance clause
  • PRA-T8 Add or justify test follow-up: Acceptance clause

Findings index

ID Severity Category Location Required action
PRA-1 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-2 Required security src/lib/actions/sandbox/process-recovery.ts:500 Restore an equivalent Hermes-only running-gateway boundary check before returning success from recover/probe/connect. If the prior root-exec implementation broke connect/recover, fix that root cause narrowly while preserving fail-closed behavior for raw-secret and inconclusive validator results.
PRA-3 Required acceptance test/process-recovery.test.ts:245 Add a minimal regression test for the actual #5530 connect/recover failure and adjust the implementation narrowly so connect/recover work while the Hermes env-file boundary remains enforced on the already-running/probe path. If a temporary compatibility workaround is unavoidable, document the invalid state, source boundary, source-fix constraint, regression test, and removal condition in code.
PRA-4 Required tests src/lib/actions/sandbox/connect-flow.test.ts:146 Restore or replace the deleted connect-flow negative tests once the implementation again exposes an explicit refusal result or throw. Keep assertions that `ensureOllamaAuthProxy`, auto-pair approval, inference-route repair, and `openshell sandbox connect` are not invoked after raw-secret or inconclusive boundary refusal.

🚨 Required before merge

Address these before merging unless a maintainer explicitly overrides the advisor with rationale.

PRA-2 Required — Restore Hermes running-gateway secret-boundary enforcement

  • Location: src/lib/actions/sandbox/process-recovery.ts:500
  • Category: security
  • Problem: The running-gateway branch in `checkAndRecoverSandboxProcesses()` now treats a live Hermes gateway as recoverable without revalidating `/sandbox/.hermes/.env`. The PR deletes the standalone validator path, the refusal result contract, the marker parsing, and the `connectSandbox()` refusal handling. The remaining Hermes guards still protect relaunch/manual recovery scripts, but they do not run when the gateway is already serving.
  • Impact: A Hermes sandbox whose env file is poisoned after startup with raw secret-shaped bot/API tokens can continue serving while `recover` or `connect --probe-only` reports success and refreshes forwards. Operators also lose the refusal diagnostics that named the offending key and remediation, creating a false healthy state for a credential-boundary violation.
  • Required action: Restore an equivalent Hermes-only running-gateway boundary check before returning success from recover/probe/connect. If the prior root-exec implementation broke connect/recover, fix that root cause narrowly while preserving fail-closed behavior for raw-secret and inconclusive validator results.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Read `src/lib/actions/sandbox/process-recovery.ts` in the `if (running)` branch and confirm there is no call to `validate-hermes-env-secret-boundary.py`, no standalone boundary check, and no `secretBoundaryRefused` result before forward/dashboard recovery returns success.
  • Missing regression test: Add a test where a persisted Hermes sandbox has a running gateway and the env-file validator reports raw secret-shaped values; assert `checkAndRecoverSandboxProcesses()` refuses, surfaces the offending validator stderr/remediation, does not refresh forwards, and does not return a successful running-gateway result.
  • Done when: The required change is committed and verification passes: Read `src/lib/actions/sandbox/process-recovery.ts` in the `if (running)` branch and confirm there is no call to `validate-hermes-env-secret-boundary.py`, no standalone boundary check, and no `secretBoundaryRefused` result before forward/dashboard recovery returns success.
  • Evidence: The diff removes `enforceHermesSecretBoundaryOnRunningGateway()`, `buildHermesEnvFileBoundaryStandaloneCheck()`, `SECRET_BOUNDARY_*` marker handling, `printValidatorStderr()`, and the `connect.ts` `exitOnSecretBoundaryRefusal()` path. Current `checkAndRecoverSandboxProcesses()` proceeds directly from `running === true` into dashboard/forward recovery.

PRA-3 Required — Add a reproducer or narrow replacement for the claimed connect/recover break

  • Location: test/process-recovery.test.ts:245
  • Category: acceptance
  • Problem: The available acceptance text says `fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 break`, `nemohermes sandbox connect.`, and `nemohermes sandbox recover...`, but the diff removes the security control wholesale and deletes the tests that described the broken boundary paths. There is no new test or code evidence identifying the invalid state that caused fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 to break connect/recover, where that state is created, or why the source cannot be fixed while retaining the boundary.
  • Impact: Without a reproducer, this PR can mask the reported operational failure by permanently removing a credential-boundary defense. Future changes can also reintroduce the connect/recover break because the root cause is not captured as executable evidence.
  • Required action: Add a minimal regression test for the actual fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 connect/recover failure and adjust the implementation narrowly so connect/recover work while the Hermes env-file boundary remains enforced on the already-running/probe path. If a temporary compatibility workaround is unavoidable, document the invalid state, source boundary, source-fix constraint, regression test, and removal condition in code.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Compare the PR body clauses with the changed tests: `src/lib/actions/sandbox/connect-flow.test.ts` and `test/process-recovery.test.ts` delete boundary-refusal cases and do not add a test that reproduces a connect/recover failure caused by the previous implementation.
  • Missing regression test: Add `recover/connect reproduce the previous fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 Hermes probe-boundary failure and preserve boundary refusal for poisoned env files`, with assertions for the exact failing connect/recover scenario and for the raw-secret negative path still refusing.
  • Done when: The required change is committed and verification passes: Compare the PR body clauses with the changed tests: `src/lib/actions/sandbox/connect-flow.test.ts` and `test/process-recovery.test.ts` delete boundary-refusal cases and do not add a test that reproduces a connect/recover failure caused by the previous implementation.
  • Evidence: The diff deletes running-Hermes tests for raw-secret refusal, missing agent definition fail-safe, unreachable root exec fail-safe, missing validator warning, and connect/probe refusal. The PR body provides only the clauses `fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 break`, `nemohermes sandbox connect.`, and `nemohermes sandbox recover...`.

PRA-4 Required — Restore negative coverage that stops connect side effects after boundary refusal

  • Location: src/lib/actions/sandbox/connect-flow.test.ts:146
  • Category: tests
  • Problem: The changed tests remove the cases that proved `connect --probe-only` and non-probe `connect` stop when the Hermes boundary refuses. The remaining connect-flow tests cover normal readiness, terminal sandbox phases, recovered gateways, and process-inspection failure, but not the security-critical branch where recovery refuses before Ollama proxy setup, inference-route repair, auto-pair approval, or SSH.
  • Impact: A future or current regression can proceed with host-side setup and SSH after a Hermes boundary refusal without any unit test catching the side effects. This weakens the caller/callee contract between `checkAndRecoverSandboxProcesses()` and `connectSandbox()` and contributes to the authentication/authorization and logging security warnings.
  • Required action: Restore or replace the deleted connect-flow negative tests once the implementation again exposes an explicit refusal result or throw. Keep assertions that `ensureOllamaAuthProxy`, auto-pair approval, inference-route repair, and `openshell sandbox connect` are not invoked after raw-secret or inconclusive boundary refusal.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Read `src/lib/actions/sandbox/connect-flow.test.ts` after the patch: only four connect-flow scenarios remain, and none configure a Hermes boundary refusal or assert that downstream connect side effects are skipped.
  • Missing regression test: Add `connectSandbox --probe-only exits non-zero and does not auto-pair or print Probe complete when Hermes running-gateway boundary refuses` and `connectSandbox non-probe exits before Ollama proxy, inference-route repair, auto-pair, and SSH when Hermes running-gateway boundary refuses`.
  • Done when: The required change is committed and verification passes: Read `src/lib/actions/sandbox/connect-flow.test.ts` after the patch: only four connect-flow scenarios remain, and none configure a Hermes boundary refusal or assert that downstream connect side effects are skipped.
  • Evidence: The diff removes three connect-flow tests named around raw-secret remediation, non-probe refusal before Ollama/inference-route/auto-pair, and inconclusive boundary guidance.
Review findings by urgency: 3 required fixes, 1 item to resolve/justify, 0 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

PRA-1 Resolve/justify — Source-of-truth review needed: Hermes recover/connect workaround reverting #5530

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as missing.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Missing. No new test reproduces the connect/recover break while preserving poisoned-env refusal.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: The diff deletes the boundary implementation and negative tests; no linked issue clauses or deterministic comments provide a root-cause reproducer beyond the PR body.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

  • None.
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — checkAndRecoverSandboxProcesses refuses a running Hermes gateway when env-file validator reports raw secret-shaped values and does not refresh forwards. The changed behavior crosses host CLI logic, OpenShell sandbox exec/SSH authority, Hermes process ownership, boundary validator execution, and port-forward recovery. Unit tests are needed for caller/callee contracts, but targeted runtime/integration validation is also warranted for the security boundary.
  • PRA-T2 Runtime validation — checkAndRecoverSandboxProcesses fails safe when running Hermes boundary validation cannot execute or returns no marker. The changed behavior crosses host CLI logic, OpenShell sandbox exec/SSH authority, Hermes process ownership, boundary validator execution, and port-forward recovery. Unit tests are needed for caller/callee contracts, but targeted runtime/integration validation is also warranted for the security boundary.
  • PRA-T3 Runtime validation — connectSandbox --probe-only exits non-zero and does not auto-pair or print Probe complete when Hermes running-gateway boundary refuses. The changed behavior crosses host CLI logic, OpenShell sandbox exec/SSH authority, Hermes process ownership, boundary validator execution, and port-forward recovery. Unit tests are needed for caller/callee contracts, but targeted runtime/integration validation is also warranted for the security boundary.
  • PRA-T4 Runtime validation — connectSandbox non-probe exits before Ollama proxy, inference-route repair, auto-pair approval, and SSH when Hermes running-gateway boundary refuses. The changed behavior crosses host CLI logic, OpenShell sandbox exec/SSH authority, Hermes process ownership, boundary validator execution, and port-forward recovery. Unit tests are needed for caller/callee contracts, but targeted runtime/integration validation is also warranted for the security boundary.
  • PRA-T5 Runtime validation — recover/connect reproduce the previous fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 connect/recover break and prove the replacement fix avoids it while preserving poisoned-env refusal. The changed behavior crosses host CLI logic, OpenShell sandbox exec/SSH authority, Hermes process ownership, boundary validator execution, and port-forward recovery. Unit tests are needed for caller/callee contracts, but targeted runtime/integration validation is also warranted for the security boundary.
  • PRA-T6 Restore negative coverage that stops connect side effects after boundary refusal — Restore or replace the deleted connect-flow negative tests once the implementation again exposes an explicit refusal result or throw. Keep assertions that `ensureOllamaAuthProxy`, auto-pair approval, inference-route repair, and `openshell sandbox connect` are not invoked after raw-secret or inconclusive boundary refusal.
  • PRA-T7 Acceptance clausefix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 break — add test evidence or identify existing coverage. No changed test or code evidence reproduces the break; the diff removes the fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 implementation and its negative tests wholesale.
  • PRA-T8 Acceptance clause — nemohermes sandbox connect. — add test evidence or identify existing coverage. `src/lib/actions/sandbox/connect.ts` changes the connect path by deleting boundary-refusal handling, but no test demonstrates the prior connect failure or proves a narrow fix preserves poisoned-env refusal.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Resolve/justify — Source-of-truth review needed: Hermes recover/connect workaround reverting #5530

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as missing.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Missing. No new test reproduces the connect/recover break while preserving poisoned-env refusal.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: The diff deletes the boundary implementation and negative tests; no linked issue clauses or deterministic comments provide a root-cause reproducer beyond the PR body.

PRA-2 Required — Restore Hermes running-gateway secret-boundary enforcement

  • Location: src/lib/actions/sandbox/process-recovery.ts:500
  • Category: security
  • Problem: The running-gateway branch in `checkAndRecoverSandboxProcesses()` now treats a live Hermes gateway as recoverable without revalidating `/sandbox/.hermes/.env`. The PR deletes the standalone validator path, the refusal result contract, the marker parsing, and the `connectSandbox()` refusal handling. The remaining Hermes guards still protect relaunch/manual recovery scripts, but they do not run when the gateway is already serving.
  • Impact: A Hermes sandbox whose env file is poisoned after startup with raw secret-shaped bot/API tokens can continue serving while `recover` or `connect --probe-only` reports success and refreshes forwards. Operators also lose the refusal diagnostics that named the offending key and remediation, creating a false healthy state for a credential-boundary violation.
  • Required action: Restore an equivalent Hermes-only running-gateway boundary check before returning success from recover/probe/connect. If the prior root-exec implementation broke connect/recover, fix that root cause narrowly while preserving fail-closed behavior for raw-secret and inconclusive validator results.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Read `src/lib/actions/sandbox/process-recovery.ts` in the `if (running)` branch and confirm there is no call to `validate-hermes-env-secret-boundary.py`, no standalone boundary check, and no `secretBoundaryRefused` result before forward/dashboard recovery returns success.
  • Missing regression test: Add a test where a persisted Hermes sandbox has a running gateway and the env-file validator reports raw secret-shaped values; assert `checkAndRecoverSandboxProcesses()` refuses, surfaces the offending validator stderr/remediation, does not refresh forwards, and does not return a successful running-gateway result.
  • Done when: The required change is committed and verification passes: Read `src/lib/actions/sandbox/process-recovery.ts` in the `if (running)` branch and confirm there is no call to `validate-hermes-env-secret-boundary.py`, no standalone boundary check, and no `secretBoundaryRefused` result before forward/dashboard recovery returns success.
  • Evidence: The diff removes `enforceHermesSecretBoundaryOnRunningGateway()`, `buildHermesEnvFileBoundaryStandaloneCheck()`, `SECRET_BOUNDARY_*` marker handling, `printValidatorStderr()`, and the `connect.ts` `exitOnSecretBoundaryRefusal()` path. Current `checkAndRecoverSandboxProcesses()` proceeds directly from `running === true` into dashboard/forward recovery.

PRA-3 Required — Add a reproducer or narrow replacement for the claimed connect/recover break

  • Location: test/process-recovery.test.ts:245
  • Category: acceptance
  • Problem: The available acceptance text says `fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 break`, `nemohermes sandbox connect.`, and `nemohermes sandbox recover...`, but the diff removes the security control wholesale and deletes the tests that described the broken boundary paths. There is no new test or code evidence identifying the invalid state that caused fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 to break connect/recover, where that state is created, or why the source cannot be fixed while retaining the boundary.
  • Impact: Without a reproducer, this PR can mask the reported operational failure by permanently removing a credential-boundary defense. Future changes can also reintroduce the connect/recover break because the root cause is not captured as executable evidence.
  • Required action: Add a minimal regression test for the actual fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 connect/recover failure and adjust the implementation narrowly so connect/recover work while the Hermes env-file boundary remains enforced on the already-running/probe path. If a temporary compatibility workaround is unavoidable, document the invalid state, source boundary, source-fix constraint, regression test, and removal condition in code.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Compare the PR body clauses with the changed tests: `src/lib/actions/sandbox/connect-flow.test.ts` and `test/process-recovery.test.ts` delete boundary-refusal cases and do not add a test that reproduces a connect/recover failure caused by the previous implementation.
  • Missing regression test: Add `recover/connect reproduce the previous fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 Hermes probe-boundary failure and preserve boundary refusal for poisoned env files`, with assertions for the exact failing connect/recover scenario and for the raw-secret negative path still refusing.
  • Done when: The required change is committed and verification passes: Compare the PR body clauses with the changed tests: `src/lib/actions/sandbox/connect-flow.test.ts` and `test/process-recovery.test.ts` delete boundary-refusal cases and do not add a test that reproduces a connect/recover failure caused by the previous implementation.
  • Evidence: The diff deletes running-Hermes tests for raw-secret refusal, missing agent definition fail-safe, unreachable root exec fail-safe, missing validator warning, and connect/probe refusal. The PR body provides only the clauses `fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path #5530 break`, `nemohermes sandbox connect.`, and `nemohermes sandbox recover...`.

PRA-4 Required — Restore negative coverage that stops connect side effects after boundary refusal

  • Location: src/lib/actions/sandbox/connect-flow.test.ts:146
  • Category: tests
  • Problem: The changed tests remove the cases that proved `connect --probe-only` and non-probe `connect` stop when the Hermes boundary refuses. The remaining connect-flow tests cover normal readiness, terminal sandbox phases, recovered gateways, and process-inspection failure, but not the security-critical branch where recovery refuses before Ollama proxy setup, inference-route repair, auto-pair approval, or SSH.
  • Impact: A future or current regression can proceed with host-side setup and SSH after a Hermes boundary refusal without any unit test catching the side effects. This weakens the caller/callee contract between `checkAndRecoverSandboxProcesses()` and `connectSandbox()` and contributes to the authentication/authorization and logging security warnings.
  • Required action: Restore or replace the deleted connect-flow negative tests once the implementation again exposes an explicit refusal result or throw. Keep assertions that `ensureOllamaAuthProxy`, auto-pair approval, inference-route repair, and `openshell sandbox connect` are not invoked after raw-secret or inconclusive boundary refusal.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Read `src/lib/actions/sandbox/connect-flow.test.ts` after the patch: only four connect-flow scenarios remain, and none configure a Hermes boundary refusal or assert that downstream connect side effects are skipped.
  • Missing regression test: Add `connectSandbox --probe-only exits non-zero and does not auto-pair or print Probe complete when Hermes running-gateway boundary refuses` and `connectSandbox non-probe exits before Ollama proxy, inference-route repair, auto-pair, and SSH when Hermes running-gateway boundary refuses`.
  • Done when: The required change is committed and verification passes: Read `src/lib/actions/sandbox/connect-flow.test.ts` after the patch: only four connect-flow scenarios remain, and none configure a Hermes boundary refusal or assert that downstream connect side effects are skipped.
  • Evidence: The diff removes three connect-flow tests named around raw-secret remediation, non-probe refusal before Ollama/inference-route/auto-pair, and inconclusive boundary guidance.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: hermes-secret-boundary-e2e
Optional E2E: hermes-e2e, sandbox-operations-e2e

Dispatch hint: hermes-secret-boundary-e2e

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • hermes-secret-boundary-e2e (medium): Closest existing E2E for Hermes credential/security boundary changes. It builds Hermes images and verifies raw secret-shaped values do not enter Hermes env/config and startup rejects introduced raw secrets.

Optional E2E

  • hermes-e2e (high): Useful adjacent confidence for the Hermes install/onboard/health/forward/live-inference path after connect/recover code changes, but it does not specifically exercise the removed running-gateway boundary refusal path and requires hosted inference secrets.
  • sandbox-operations-e2e (high): Covers real sandbox lifecycle, connect/chat, status-driven process recovery, and gateway recovery for OpenClaw. It is adjacent to process-recovery/connect changes but not Hermes-specific.

New E2E recommendations

  • Hermes recovery secret boundary (high): Existing E2Es cover Hermes image/startup secret-boundary behavior but do not appear to exercise nemohermes <sandbox> recover or connect --probe-only against an already-running Hermes gateway after /sandbox/.hermes/.env has been poisoned with a raw secret-shaped value. The changed code removes that exact enforcement path, so a dedicated E2E would catch regressions or intentional policy changes in the real sandbox lifecycle.
    • Suggested test: Add a Hermes recovery secret-boundary E2E that onboards a Hermes sandbox, injects a raw secret-shaped value into /sandbox/.hermes/.env while the gateway is running, runs nemohermes <name> recover or connect --probe-only, and asserts the documented accept/refuse behavior and gateway health outcome.

Dispatch hint

  • Workflow: nightly-e2e.yaml
  • jobs input: hermes-secret-boundary-e2e

@github-actions

github-actions Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: issue-2478-crash-loop-recovery-vitest
Optional Vitest E2E scenarios: hermes-inference-switch-vitest

Dispatch required Vitest E2E scenarios:

  • gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field jobs=issue-2478-crash-loop-recovery-vitest

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: medium

Required Vitest E2E scenarios

  • issue-2478-crash-loop-recovery-vitest: Changes touch checkAndRecoverSandboxProcesses and connect --probe-only recovery flow; this job repeatedly exercises the production recovery path through live Vitest.
    • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field jobs=issue-2478-crash-loop-recovery-vitest

Optional Vitest E2E scenarios

  • hermes-inference-switch-vitest: Adjacent Hermes live coverage for a real Hermes sandbox with inference switching; useful if extra confidence is needed beyond the primary Hermes runtime and recovery jobs.
    • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field jobs=hermes-inference-switch-vitest

Relevant changed files

  • src/lib/actions/sandbox/connect.ts
  • src/lib/actions/sandbox/process-recovery.ts
  • src/lib/agent/hermes-recovery-boundary.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/actions/sandbox/process-recovery.ts`:
- Line 490: The fail-closed Hermes boundary check was removed from the
live-gateway recovery path, allowing dashboard and forward recovery for
already-running gateways to proceed without re-validating the Hermes environment
file at `/sandbox/.hermes/.env`. This can leave a poisoned Hermes environment in
place without proper validation failure. Restore the deleted running-gateway
validator check or add an equivalent fail-closed boundary gate before any
dashboard or forward recovery operations in the affected code block (around line
506 and throughout the recovery logic in lines 502-568) to ensure the Hermes
environment is properly validated and `SECRET_BOUNDARY_REFUSED` is emitted when
appropriate.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 066c32a2-d2ec-4ac6-a269-3daf3c6e9e4b

📥 Commits

Reviewing files that changed from the base of the PR and between 9cb773e and 91d51db.

📒 Files selected for processing (8)
  • docs/reference/commands-nemohermes.mdx
  • docs/reference/commands.mdx
  • src/lib/actions/sandbox/connect-flow.test.ts
  • src/lib/actions/sandbox/connect.ts
  • src/lib/actions/sandbox/process-recovery.ts
  • src/lib/agent/hermes-recovery-boundary.ts
  • src/lib/agent/runtime-hermes-secret-boundary-behavioural.test.ts
  • test/process-recovery.test.ts
💤 Files with no reviewable changes (5)
  • docs/reference/commands-nemohermes.mdx
  • docs/reference/commands.mdx
  • src/lib/actions/sandbox/connect-flow.test.ts
  • src/lib/agent/runtime-hermes-secret-boundary-behavioural.test.ts
  • src/lib/agent/hermes-recovery-boundary.ts

Comment thread src/lib/actions/sandbox/process-recovery.ts
@sandl99 sandl99 changed the title Revert "fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path" fix(sandbox/recover): Revert "enforce Hermes env-file secret boundary on probe path" Jun 19, 2026
Signed-off-by: San Dang <sdang@nvidia.com>
@sandl99 sandl99 changed the title fix(sandbox/recover): Revert "enforce Hermes env-file secret boundary on probe path" fix(sandbox/recover): revert Hermes probe boundary enforcement Jun 22, 2026
Signed-off-by: San Dang <sdang@nvidia.com>

# Conflicts:
#	test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
@sandl99
sandl99 requested a review from cv June 22, 2026 02:31
@sandl99 sandl99 self-assigned this Jun 22, 2026
@sandl99 sandl99 added bug bug-fix PR fixes a bug or regression labels Jun 22, 2026
@sandl99

sandl99 commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator Author

Currently #5530 blocked me from testing new feature with nemohermes connect or recover. Please feel free add a fix or approve this revert PR to fix the issue. @cv @laitingsheng

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reopened #5525 since its security gap returns with the revert; folding the operational break into that ticket since Carlos is taking the narrow fix

@cv

cv commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Thanks @sandl99 — I checked main after #5530 and looks like we have not landed a replacement/narrow fix yet. I don’t think we should merge this full revert as-is because it reopens #5525: recover / connect --probe-only would again report success for an already-running Hermes gateway even when /sandbox/.hermes/.env` contains raw secret-shaped values.

Can you paste the exact redacted stderr from:

nemohermes <sandbox> connect --probe-only
nemohermes <sandbox> recover

Please redact values, but keep the key names and the boundary/refusal/inconclusive lines.

If the failure is a raw-secret refusal, the right fix is probably to update the new feature’s Hermes config path to write openshell:resolve:env: placeholders instead of raw token values. If it is an inconclusive validator/root-exec failure, I’ll put up a narrow fix that preserves the #5530 security boundary instead of reverting it wholesale.

@sandl99

sandl99 commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator Author

@cv @cjagwani Sorry I forgot to put the stderr in the description. Put it now. I am good if we have a forward fix for that given it's security issue.

NemoClaw git:(main) ✗ nemohermes sb connect      

  ⚠ Sandbox 'sb' is running Hermes Agent 0.14.0 (current: 2026.5.16)
    Run: nemoclaw sb rebuild


  Secret-boundary check could not run against the Hermes gateway in 'sb'.
  Refusing recovery to keep the validator-enforced boundary intact.

  Connect failed: secret-boundary check did not complete for Hermes Agent gateway in 'sb'.
  Inspect the validator output above and re-run `nemoclaw <sandbox> recover`.

@sandl99

sandl99 commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator Author

@cv @cjagwani (Re-comment as my previous replies suddenly disappear)
I'm sorry as I forgot to put stderr into description. Here is stderr when I run nemohermes on latest code.
I'm good for a forward fix as well given it's security issue.

NemoClaw git:(main) ✗ nemohermes sb connect      

  ⚠ Sandbox 'sb' is running Hermes Agent 0.14.0 (current: 2026.5.16)
    Run: nemoclaw sb rebuild


  Secret-boundary check could not run against the Hermes gateway in 'sb'.
  Refusing recovery to keep the validator-enforced boundary intact.

  Connect failed: secret-boundary check did not complete for Hermes Agent gateway in 'sb'.
  Inspect the validator output above and re-run `nemoclaw <sandbox> recover`.

@cv

cv commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Thanks @sandl99 — I opened a forward-fix PR that keeps the #5530 Hermes running-gateway boundary enforcement but makes the root openshell sandbox exec parser tolerate the stdout framing that appears to be triggering your inconclusive failure:

Could you please try that branch against your sb sandbox when you have a chance?

git fetch origin pull/5590/head:pr-5590-hermes-boundary-framing
git switch pr-5590-hermes-boundary-framing
npm run build:cli
nemohermes sb connect
nemohermes sb recover

Expected result: the normal healthy sandbox should no longer fail with Secret-boundary check could not run.... If you can also test a poisoned /sandbox/.hermes/.env, it should still refuse and show the offending key rather than silently proceeding.

@sandl99

sandl99 commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator Author

@cv I still encountered same issue after running on your bits. It can be repro in my both Ubuntu and MacOS

sdang@sdangvm01:~/NVIDIA/NemoClaw$ nemohermes hm connect

  Secret-boundary check could not run against the Hermes gateway in 'hm'.
  Refusing recovery to keep the validator-enforced boundary intact.

  Connect failed: secret-boundary check did not complete for Hermes Agent gateway in 'hm'.
  Inspect the validator output above and re-run `nemoclaw <sandbox> recover`.

@cv

cv commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Thanks @sandl99 — agreed, that means my first parser-only fix was incomplete.

I couldn't reproduce with a live sandbox locally because my OpenShell gateway is down here (openshell sandbox list returns a transport connection refused), but I dug into the failing path and updated #5590 with two more fixes that match this symptom:

  1. The Hermes validator root-exec payload is now encoded before sh -c, so OpenShell does not receive the multi-line standalone boundary script directly.
  2. If openshell sandbox exec still does not produce the NemoClaw startup marker, Docker-backed local sandboxes now fall back to docker exec -u root against openshell-<sandbox> before failing closed.

Could you please retry #5590 at latest (7cfa0dbf8 or newer) on hm?

git fetch origin fix/hermes-recover-root-exec-framing
git switch fix/hermes-recover-root-exec-framing
git pull --ff-only
npm run build:cli
nemohermes hm connect
nemohermes hm recover

If it still fails, the next useful datapoint would be whether this command prints the marker when run directly:

openshell sandbox exec --name hm -- sh -c "printf '%s\\n' '__NEMOCLAW_SANDBOX_EXEC_STARTED__'; echo SECRET_BOUNDARY_OK"

@sandl99

sandl99 commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator Author

@cv, it works perfectly at 7cfa0db and 2829b1d (some CI bugs). You can wrap it up at 7cfa0db. Thank you.

sdang@sdangvm01:~/NVIDIA/NemoClaw$ nemohermes hm connect

  ✓ Connecting to sandbox 'hm'
  Inside the sandbox, run `hermes` to start chatting with the agent.
  Type `/exit` to leave the chat, then `exit` to return to the host shell.

sandbox@2f9e1e493158:~$ 
exit
sdang@sdangvm01:~/NVIDIA/NemoClaw$ nemohermes hm recover
  Probe complete: Hermes Agent gateway is running in 'hm'; restored dashboard port forward.

@sandl99 sandl99 closed this Jun 22, 2026
@cv

cv commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the fast retest @sandl99. Since your validation passed at 7cfa0dbf8, I documented the decision on #5590 to keep the Docker root-exec fallback for this hotfix rather than removing it to satisfy the advisor: #5590 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix PR fixes a bug or regression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants