Skip to content

fix: repair nightly Hermes and shields regressions - #2877

Merged
ericksoa merged 7 commits into
mainfrom
fix/nightly-hermes-shields-regressions
May 2, 2026
Merged

fix: repair nightly Hermes and shields regressions#2877
ericksoa merged 7 commits into
mainfrom
fix/nightly-hermes-shields-regressions

Conversation

@ericksoa

@ericksoa ericksoa commented May 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Make Hermes onboarding accept the configured absolute binary path when PATH-based command resolution is unavailable, while keeping a stable Hermes PATH in pre-created rc files.
  • Restore the OpenClaw mutable config contract in non-root startup with usable group read/write/execute permissions and setgid directories.
  • Clear setgid during shields-up locking so locked config directories verify as 755, and update shields E2E expectations to 660/2770 mutable mode.

Validation

  • npm ci --ignore-scripts
  • npm run build:cli
  • npm test -- test/repro-2376.test.ts src/lib/agent-onboard.test.ts test/shields.test.ts test/repro-2681-group-writable.test.ts test/nemoclaw-start.test.ts
  • bash -n scripts/nemoclaw-start.sh test/e2e/test-shields-config.sh
  • npm run format:check
  • npm run lint

Notes

Summary by CodeRabbit

  • Bug Fixes

    • Fixed shell initialization to properly include virtual environment paths in PATH
    • Enhanced configuration permission enforcement with stricter security controls
  • Improvements

    • Refined configuration lock and unlock behavior for improved security handling
    • Updated mutable configuration default permissions to use stricter access controls with group write protection and directory inheritance
    • Improved configuration synchronization and permission normalization in sandbox environments

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@ericksoa ericksoa self-assigned this May 2, 2026
@coderabbitai

coderabbitai Bot commented May 2, 2026

Copy link
Copy Markdown
Contributor

Caution

Review failed

Pull request was closed or merged during review

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR standardizes file permissions for the sandbox mutable configuration directory from a mixed scheme to a consistent setgid-enabled group-writable model (2770 on directories, 660 on files). It also adds Hermes venv binaries to the shell initialization PATH, and refactors permission-locking logic to conditionally strip setgid during config immutability.

Changes

Shell Environment & PATH Setup

Layer / File(s) Summary
Shell Initialization
agents/hermes/Dockerfile.base
.bashrc and .profile now export PATH="/usr/local/bin:/opt/hermes/.venv/bin:${PATH}" after sourcing the proxy-env script.
Tests
test/repro-2376.test.ts
New assertions verify that rc files prepend Hermes paths to PATH and support configurable commands beyond hardcoded printf.

Mutable Config Permission Standardization

Layer / File(s) Summary
Documentation
Dockerfile, src/lib/shields.ts
Updated comments clarify mutable default is 2770/660 with setgid; immutability is opt-in via shields up (444 root:root + chattr +i).
Permission Normalization Functions
scripts/nemoclaw-start.sh, src/lib/onboard.ts
normalize_mutable_config_perms() and buildSandboxConfigSyncScript() now recursively apply chmod g+rwX,o-rwx, enforce setgid on directories via find -exec chmod g+s, and explicitly set 2770 on config dir and 660 on config files.
Lock/Unlock Behavior
src/lib/shields.ts
applyStateDirLockMode() refactored to parametrize lock/unlock modes and conditionally clear setgid (chmod g-s) only when locking. lockAgentConfig() now clears setgid on root config dir post-locking and restores it to plain 755.
Default Permissions
Dockerfile
/sandbox/.openclaw initial permissions changed from mixed scheme (chmod 755, chmod 644, chmod -R g+w) to chmod -R g+rwX,o-rwx, setgid on dirs, chmod 2770, and chmod 660 on files; .config-hash mode updated from 664 to 660.
Startup Scripts
scripts/nemoclaw-start.sh
Non-root startup path now invokes normalize_mutable_config_perms() after config verification and ownership repair; fix_openclaw_ownership() applies 2770 to directory and 660 to files.
Assertion Updates
test/e2e/test-shields-config.sh, test/nemoclaw-start.test.ts, test/repro-2681-group-writable.test.ts, test/shields.test.ts, test/onboard.test.ts
Tests updated to expect and validate 2770 directory mode with setgid and 660 file mode; new assertions verify conditional setgid-stripping logic and workspace permission inheritance during lock/unlock cycles.

Sequence Diagram(s)

sequenceDiagram
    participant Startup as Non-Root Startup
    participant ConfigVerify as Config Verification
    participant Normalize as normalize_mutable_config_perms()
    participant Lock as lockAgentConfig()
    participant FSState as Filesystem State

    Startup->>ConfigVerify: verify_config_integrity_if_locked()
    ConfigVerify->>FSState: Check /sandbox/.openclaw
    ConfigVerify->>Startup: Integrity valid or repaired
    
    Startup->>Normalize: normalize_mutable_config_perms()
    Normalize->>FSState: chmod -R g+rwX,o-rwx
    Normalize->>FSState: find -exec chmod g+s (dirs)
    Normalize->>FSState: chmod 2770 /sandbox/.openclaw
    Normalize->>FSState: chmod 660 on config files
    Normalize->>Startup: Permissions normalized

    Startup->>Lock: shields up (on demand)
    Lock->>FSState: applyStateDirLockMode (locking=true)
    Lock->>FSState: chmod g-s /sandbox/.openclaw (clear setgid)
    Lock->>FSState: chmod 755 /sandbox/.openclaw
    Lock->>Startup: Config locked (immutable)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

The PR spans multiple interconnected subsystems (shell initialization, permission normalization, lock/unlock semantics) with heterogeneous logic changes. The permission model refactoring requires careful validation across startup paths, container defaults, and lock state transitions. Test expansions are substantial and logic-dense, particularly for conditional setgid handling and workspace inheritance patterns.

Poem

A rabbit hops through permissions fair,
Setting modes with utmost care,
2770 for dirs with setgid's grace,
660 files in sandbox's place,
Locking, unlocking, bit by bit,
The config garden is freshly knit! 🐰✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the primary change: fixing regressions in Hermes and shields functionality, which aligns with the multiple file changes addressing onboarding, permissions, and lock/unlock behavior across the codebase.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/nightly-hermes-shields-regressions

Review rate limit: 9/10 reviews remaining, refill in 6 minutes.

Comment @coderabbitai help to get the list of available commands and usage tips.

ericksoa added 2 commits May 1, 2026 19:07
…ields-regressions

# Conflicts:
#	test/repro-2681-group-writable.test.ts
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/lib/shields.ts (1)

270-318: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Restore the same mutable-default mode that startup uses.

The unlock branch still applies 2775 plus recursive g+w,o-w, so a shields up/down cycle does not converge back to the 660/2770 + o-rwx contract from scripts/nemoclaw-start.sh. Top-level state dirs become world-traversable, and descendants can retain stale read/execute bits or miss g+r entirely depending on their pre-lock mode. That reopens credentials/identity/workspace more broadly than intended after shields down.

Suggested fix
-  const writeStrip = isLocking ? "go-w" : "g+w,o-w";
-  const dirMode = isLocking ? "755" : "2775";
+  const recursiveMode = isLocking ? "go-w" : "g+rwX,o-rwx";
+  const dirMode = isLocking ? "755" : "2770";
@@
-      kubectlExec(sandboxName, ["chmod", "-R", writeStrip, dirPath]);
+      kubectlExec(sandboxName, ["chmod", "-R", recursiveMode, dirPath]);
@@
-write_strip="$3"
+recursive_mode="$3"
 dir_mode="$4"
 clear_setgid="$5"
 for dir in "$config_dir"/workspace-*; do
   [ -d "$dir" ] || continue
   chown -R "$owner" "$dir" 2>/dev/null || true
   chmod "$dir_mode" "$dir" 2>/dev/null || true
   [ "$clear_setgid" = "1" ] && chmod g-s "$dir" 2>/dev/null || true
-  chmod -R "$write_strip" "$dir" 2>/dev/null || true
+  chmod -R "$recursive_mode" "$dir" 2>/dev/null || true
 done
@@
-      writeStrip,
+      recursiveMode,
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/lib/shields.ts` around lines 270 - 318, The unlock branch uses writeStrip
= "g+w,o-w" and dirMode = "2775", which doesn't restore the startup permissions;
change the unlock values so they match the startup contract (top-level dirs 2770
and recursive owner/group read/write with others removed). Specifically, update
the declarations to use writeStrip = isLocking ? "go-w" : "g+rw,o-rwx" and
dirMode = isLocking ? "755" : "2770", and ensure the same values are passed to
kubectlExec in both the HIGH_RISK_STATE_DIRS loop (kubectlExec calls) and the
workspace loop (the heredoc args configDir, owner, writeStrip, dirMode,
clearSetgid) so post-unlock permissions converge to 660/2770 + o-rwx as startup
expects.
🧹 Nitpick comments (1)
agents/hermes/Dockerfile.base (1)

118-123: Please run the Hermes selective E2E workflows before merge.

Given this is under agents/hermes/**, running hermes-e2e and rebuild-hermes-e2e would reduce regression risk for onboarding/health/inference paths.

As per coding guidelines: agents/hermes/** → E2E recommendation includes hermes-e2e and rebuild-hermes-e2e.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@agents/hermes/Dockerfile.base` around lines 118 - 123, Change requires
running the Hermes selective end-to-end workflows before merging: execute the
hermes-e2e and rebuild-hermes-e2e workflows to validate onboarding, health, and
inference paths affected by the Dockerfile.base changes; if failures occur, fix
the Dockerfile changes (the bashrc PATH/export block) and re-run these two
workflows until they pass, then include passing workflow run IDs in the PR
before merge.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@src/lib/shields.ts`:
- Around line 270-318: The unlock branch uses writeStrip = "g+w,o-w" and dirMode
= "2775", which doesn't restore the startup permissions; change the unlock
values so they match the startup contract (top-level dirs 2770 and recursive
owner/group read/write with others removed). Specifically, update the
declarations to use writeStrip = isLocking ? "go-w" : "g+rw,o-rwx" and dirMode =
isLocking ? "755" : "2770", and ensure the same values are passed to kubectlExec
in both the HIGH_RISK_STATE_DIRS loop (kubectlExec calls) and the workspace loop
(the heredoc args configDir, owner, writeStrip, dirMode, clearSetgid) so
post-unlock permissions converge to 660/2770 + o-rwx as startup expects.

---

Nitpick comments:
In `@agents/hermes/Dockerfile.base`:
- Around line 118-123: Change requires running the Hermes selective end-to-end
workflows before merging: execute the hermes-e2e and rebuild-hermes-e2e
workflows to validate onboarding, health, and inference paths affected by the
Dockerfile.base changes; if failures occur, fix the Dockerfile changes (the
bashrc PATH/export block) and re-run these two workflows until they pass, then
include passing workflow run IDs in the PR before merge.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7fa90049-7c12-4dc8-a08b-574c51859bc1

📥 Commits

Reviewing files that changed from the base of the PR and between 5474e79 and 5640fd2.

📒 Files selected for processing (10)
  • agents/hermes/Dockerfile.base
  • scripts/nemoclaw-start.sh
  • src/lib/agent-onboard.test.ts
  • src/lib/agent-onboard.ts
  • src/lib/shields.ts
  • test/e2e/test-shields-config.sh
  • test/nemoclaw-start.test.ts
  • test/repro-2376.test.ts
  • test/repro-2681-group-writable.test.ts
  • test/shields.test.ts

@ericksoa
ericksoa requested a review from cv May 2, 2026 02:25
@github-actions

github-actions Bot commented May 2, 2026

Copy link
Copy Markdown
Contributor

Selective E2E Results — ❌ Some jobs failed

Run: 25242756784
Branch: fix/nightly-hermes-shields-regressions
Requested jobs: shields-config-e2e
Summary: 0 passed, 1 failed, 21 skipped

Job Result
cloud-e2e ⏭️ skipped
cloud-inference-e2e ⏭️ skipped
cloud-onboard-e2e ⏭️ skipped
deployment-services-e2e ⏭️ skipped
diagnostics-e2e ⏭️ skipped
docs-validation-e2e ⏭️ skipped
gpu-e2e ⏭️ skipped
hermes-e2e ⏭️ skipped
inference-routing-e2e ⏭️ skipped
messaging-compatible-endpoint-e2e ⏭️ skipped
messaging-providers-e2e ⏭️ skipped
network-policy-e2e ⏭️ skipped
overlayfs-autofix-e2e ⏭️ skipped
rebuild-hermes-e2e ⏭️ skipped
rebuild-openclaw-e2e ⏭️ skipped
sandbox-operations-e2e ⏭️ skipped
sandbox-survival-e2e ⏭️ skipped
shields-config-e2e ❌ failure
skill-agent-e2e ⏭️ skipped
snapshot-commands-e2e ⏭️ skipped
token-rotation-e2e ⏭️ skipped
upgrade-stale-sandbox-e2e ⏭️ skipped

Failed jobs: shields-config-e2e. Check run artifacts for logs.

@github-actions

github-actions Bot commented May 2, 2026

Copy link
Copy Markdown
Contributor

Selective E2E Results — ✅ All requested jobs passed

Run: 25243200348
Branch: fix/nightly-hermes-shields-regressions
Requested jobs: shields-config-e2e
Summary: 1 passed, 0 failed, 21 skipped

Job Result
cloud-e2e ⏭️ skipped
cloud-inference-e2e ⏭️ skipped
cloud-onboard-e2e ⏭️ skipped
deployment-services-e2e ⏭️ skipped
diagnostics-e2e ⏭️ skipped
docs-validation-e2e ⏭️ skipped
gpu-e2e ⏭️ skipped
hermes-e2e ⏭️ skipped
inference-routing-e2e ⏭️ skipped
messaging-compatible-endpoint-e2e ⏭️ skipped
messaging-providers-e2e ⏭️ skipped
network-policy-e2e ⏭️ skipped
overlayfs-autofix-e2e ⏭️ skipped
rebuild-hermes-e2e ⏭️ skipped
rebuild-openclaw-e2e ⏭️ skipped
sandbox-operations-e2e ⏭️ skipped
sandbox-survival-e2e ⏭️ skipped
shields-config-e2e ✅ success
skill-agent-e2e ⏭️ skipped
snapshot-commands-e2e ⏭️ skipped
token-rotation-e2e ⏭️ skipped
upgrade-stale-sandbox-e2e ⏭️ skipped

…ields-regressions

# Conflicts:
#	src/lib/agent-onboard.test.ts
#	src/lib/agent-onboard.ts
#	src/lib/shields.ts
#	test/e2e/test-shields-config.sh
@ericksoa
ericksoa enabled auto-merge (squash) May 2, 2026 04:13
@ericksoa
ericksoa merged commit 7d0c2e8 into main May 2, 2026
14 of 15 checks passed
@wscurran wscurran added bug-fix PR fixes a bug or regression and removed fix labels Jun 3, 2026
@cv
cv deleted the fix/nightly-hermes-shields-regressions branch June 28, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix PR fixes a bug or regression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants