fix(e2e): encode multiline sandbox scripts - #6013
Conversation
Signed-off-by: Carlos Villela <cvillela@nvidia.com> (cherry picked from commit 9a280f4)
📝 WalkthroughWalkthrough
Sandbox shell argument encoding
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Suggested labels
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in the Show a code coverage summary of the most covered files.
TypeScript / code-coverage/cliThe overall coverage in the Show a code coverage summary of the most covered files.
Updated |
|
Runtime evidence and acceptance gate:
Do not treat the unit/CI gates alone as parity proof. This PR should merge only after both OpenClaw and Hermes jobs pass and their structured artifacts are inspected for the security-posture summary, cleanup failures, and credential-shaped leakage. |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
test/e2e-scenario/support-tests/e2e-clients.test.ts (1)
412-416: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winExercise the generated argument through
sh -lc.The regex +
Bufferround-trip proves encoding shape, but not that the emitted shell argument actually runs at the shell boundary. Add an observable execution assertion for the captured argument so this test catches quoting/decoder portability regressions.Suggested test strengthening
const encoded = argument.match(/'([A-Za-z0-9+/=]+)' \| base64 -d/u)?.[1]; expect(encoded).toBeTruthy(); expect(Buffer.from(encoded ?? "", "base64").toString("utf8")).toBe(source); + const { stdout } = await execFileAsync("sh", ["-lc", argument]); + expect(stdout).toBe("ready\n");Define
execFileAsyncwith Node’schild_process.execFile+util.promisifynear the existing test helpers/imports.As per path instructions, “Review tests for behavioral confidence rather than implementation lock-in” and “Preserve real shell, process, installer, platform, and full-journey boundaries by invoking them from Vitest when they are the contract.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e-scenario/support-tests/e2e-clients.test.ts` around lines 412 - 416, The generated argument is only being checked for encoding shape and round-trip, so the test in e2e-clients.test.ts should also execute the captured shell argument through a real shell boundary. Add a small helper using execFileAsync (built from child_process.execFile and util.promisify) near the existing test utilities, then in the relevant assertion block run the captured argument with sh -lc and verify it produces the expected source output. Keep the existing regex and Buffer checks, but make the test assert observable execution behavior from the emitted argument.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@test/e2e-scenario/support-tests/e2e-clients.test.ts`:
- Around line 412-416: The generated argument is only being checked for encoding
shape and round-trip, so the test in e2e-clients.test.ts should also execute the
captured shell argument through a real shell boundary. Add a small helper using
execFileAsync (built from child_process.execFile and util.promisify) near the
existing test utilities, then in the relevant assertion block run the captured
argument with sh -lc and verify it produces the expected source output. Keep the
existing regex and Buffer checks, but make the test assert observable execution
behavior from the emitted argument.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 1210b509-6cd9-4629-b435-29af4034632c
📒 Files selected for processing (2)
test/e2e-scenario/fixtures/clients/sandbox.tstest/e2e-scenario/support-tests/e2e-clients.test.ts
PR Review Advisor (Nemotron Ultra) — No blocking findingsMerge posture: No blocking advisor findings Action checklist
Findings index
Review findings by urgency: 0 required fixes, 1 item to resolve/justify, 2 in-scope improvements
|
E2E Advisor RecommendationRequired E2E: None Dispatch hint: Full advisor summaryE2E Recommendation AdvisorBase: Required E2E
Optional E2E
New E2E recommendations
Dispatch hint
|
Vitest E2E Scenario RecommendationRequired Vitest E2E scenarios: Dispatch required Vitest E2E scenarios:
Full Vitest E2E advisor summaryVitest E2E Scenario AdvisorBase: Required Vitest E2E scenarios
Optional Vitest E2E scenarios
Relevant changed files
|
PR Review Advisor — No blocking findingsMerge posture: No blocking advisor findings This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision. |
Vitest E2E Scenario Results — ✅ All requested jobs passedRun: 28400170844
|
|
Corrected live acceptance is complete for commit Artifact inspection (not just job conclusions) confirms for both agents:
The artifacts also explicitly record the current PID 1 posture ( This closes the runtime regression introduced by passing multiline scripts as an OpenShell command argument. Ordinary PR CI is also green with no unresolved automated-review threads. Independent human review remains the merge gate. |
<!-- markdownlint-disable MD041 --> ## Summary Retires the remaining legacy shell-driven E2E lanes for #5098 Phase 11 and makes the surviving E2E surface a single target/live Vitest workflow. This terminal cleanup deletes the old runner paths, consolidates fixtures and workflow controls, and preserves the operational, docs-validation, and two-agent security-posture coverage landed while the migration was in flight. ## Related Issues - Parent migration: #5098 - Cutover acceptance and post-merge burn-in: #5919 ## Changes - Replace `.github/workflows/e2e-vitest-scenarios.yaml` with `.github/workflows/e2e.yaml`; remove the legacy nightly/script workflows and shared script action. - Move E2E fixtures, live tests, registry, manifests, support tests, and migration docs under `test/e2e/` with target/live naming. - Delete converted `test/e2e/test-*.sh` entrypoints and the retired shell-runner test/support code while retaining implementation shell fixtures used by Vitest. - Rename target advisor files/schema under `tools/e2e-advisor/`, remove auto-dispatch, and align PR Review Advisor terminology. - Preserve #6012 controls: inventory-derived selection, fail-closed selectors, complete aggregation, PR reporting, scheduled failure issues, timing sanitization, and scorecards. - Preserve #6013 behavior: multiline sandbox scripts plus default-enabled `docs-validation` and two-agent `security-posture` jobs in the final workflow/path/project model. - Keep parity decisions and execution evidence in #5919; exact-head full, explicit-only, malformed-selector, and selective-dispatch runs are linked there. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check all that apply. For any "covered by existing tests", "not applicable", or waiver entry, add a brief justification on the same line or in the Changes section. --> - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: exact-head CodeQL/GHAS and #5919 domain approvals are pending - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: CodeRabbit could not perform a line review because the 409-file cutover exceeds its 300-file service limit; exact-head GPT-5.5 and Nemotron findings are dispositioned in PR comments, while human #5919 domain review remains pending ## Verification <!-- Check each item you ran and confirmed. Leave unchecked items you skipped. Doc-only changes do not require npm test unless you ran it. --> - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Git hooks passed during commit and push, or `npx prek run --from-ref main --to-ref HEAD` passes - [x] Targeted tests pass for changed behavior - [x] Full `npm test` passes (broad runtime changes only) - [ ] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — passed with 0 errors and 2 existing Fern warnings - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Carlos Villela <cvillela@nvidia.com> --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
<!-- markdownlint-disable MD041 --> ## Summary Fix the shared Vitest E2E sandbox client after the exact-head security-posture run proved that current OpenShell rejects newline-bearing command arguments. Multiline trusted scripts are now base64-encoded into one shell argument at the client boundary instead of requiring every scenario to invent its own minifier. ## Related Issue Contributes to NVIDIA#5919. Follow-up to merged NVIDIA#6010 and failed live run [28397462103](https://github.com/NVIDIA/NemoClaw/actions/runs/28397462103). ## Changes - Preserve direct single-line `sh -lc` arguments. - Encode multiline trusted scripts as a single newline-free `eval` argument and decode inside the sandbox. - Reject NUL-bearing scripts before process construction. - Add a round-trip contract test for the exact command argument. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: internal E2E transport compatibility only; no user-facing product behavior changes. - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: pending independent review. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Git hooks passed during commit and push, or `npx prek run --from-ref main --to-ref HEAD` passes - [x] Targeted tests pass for changed behavior - [ ] Full `npm test` passes (broad runtime changes only) - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) Targeted verification: - `VITEST_MAX_WORKERS=16 npx vitest run --project e2e-vitest-support test/e2e-scenario/support-tests/e2e-clients.test.ts test/e2e-scenario/support-tests/security-posture.test.ts` - `npm run typecheck:cli` - `make check` - normal signed commit and push hooks --- Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved handling of trusted sandbox shell scripts so multiline scripts are safely passed without raw line breaks. * Added stricter script validation to reject empty scripts and scripts containing NUL characters. * Updated shell execution behavior to reduce the risk of malformed command arguments when running scripts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Carlos Villela <cvillela@nvidia.com>
<!-- markdownlint-disable MD041 --> ## Summary Retires the remaining legacy shell-driven E2E lanes for NVIDIA#5098 Phase 11 and makes the surviving E2E surface a single target/live Vitest workflow. This terminal cleanup deletes the old runner paths, consolidates fixtures and workflow controls, and preserves the operational, docs-validation, and two-agent security-posture coverage landed while the migration was in flight. ## Related Issues - Parent migration: NVIDIA#5098 - Cutover acceptance and post-merge burn-in: NVIDIA#5919 ## Changes - Replace `.github/workflows/e2e-vitest-scenarios.yaml` with `.github/workflows/e2e.yaml`; remove the legacy nightly/script workflows and shared script action. - Move E2E fixtures, live tests, registry, manifests, support tests, and migration docs under `test/e2e/` with target/live naming. - Delete converted `test/e2e/test-*.sh` entrypoints and the retired shell-runner test/support code while retaining implementation shell fixtures used by Vitest. - Rename target advisor files/schema under `tools/e2e-advisor/`, remove auto-dispatch, and align PR Review Advisor terminology. - Preserve NVIDIA#6012 controls: inventory-derived selection, fail-closed selectors, complete aggregation, PR reporting, scheduled failure issues, timing sanitization, and scorecards. - Preserve NVIDIA#6013 behavior: multiline sandbox scripts plus default-enabled `docs-validation` and two-agent `security-posture` jobs in the final workflow/path/project model. - Keep parity decisions and execution evidence in NVIDIA#5919; exact-head full, explicit-only, malformed-selector, and selective-dispatch runs are linked there. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check all that apply. For any "covered by existing tests", "not applicable", or waiver entry, add a brief justification on the same line or in the Changes section. --> - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: exact-head CodeQL/GHAS and NVIDIA#5919 domain approvals are pending - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: CodeRabbit could not perform a line review because the 409-file cutover exceeds its 300-file service limit; exact-head GPT-5.5 and Nemotron findings are dispositioned in PR comments, while human NVIDIA#5919 domain review remains pending ## Verification <!-- Check each item you ran and confirmed. Leave unchecked items you skipped. Doc-only changes do not require npm test unless you ran it. --> - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Git hooks passed during commit and push, or `npx prek run --from-ref main --to-ref HEAD` passes - [x] Targeted tests pass for changed behavior - [x] Full `npm test` passes (broad runtime changes only) - [ ] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — passed with 0 errors and 2 existing Fern warnings - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Carlos Villela <cvillela@nvidia.com> --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Summary
Fix the shared Vitest E2E sandbox client after the exact-head security-posture run proved that current OpenShell rejects newline-bearing command arguments. Multiline trusted scripts are now base64-encoded into one shell argument at the client boundary instead of requiring every scenario to invent its own minifier.
Related Issue
Contributes to #5919. Follow-up to merged #6010 and failed live run 28397462103.
Changes
sh -lcarguments.evalargument and decode inside the sandbox.Type of Change
Quality Gates
Verification
Verifiedin GitHubnpx prek run --from-ref main --to-ref HEADpassesnpm testpasses (broad runtime changes only)npm run docsbuilds without warnings (doc changes only)Targeted verification:
VITEST_MAX_WORKERS=16 npx vitest run --project e2e-vitest-support test/e2e-scenario/support-tests/e2e-clients.test.ts test/e2e-scenario/support-tests/security-posture.test.tsnpm run typecheck:climake checkSigned-off-by: Carlos Villela cvillela@nvidia.com
Summary by CodeRabbit