Skip to content

fix(e2e): avoid multiline Deep Agents secret probe #5902

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cv merged 3 commits into from
Jun 27, 2026
Merged

fix(e2e): avoid multiline Deep Agents secret probe #5902

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ca55a72
fix(e2e): avoid multiline Deep Agents secret probe
cv Jun 27, 2026
8a63f84
test(e2e): cover Deep Agents secret probe shape
cv Jun 27, 2026
b7bfba2
test(e2e): narrow Deep Agents secret probes
cv Jun 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions test/e2e-scenario/support-tests/platform-parity-cloud-experimental.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,28 @@ describe("P0-E cloud-experimental parity guardrails", () => {
expect(result.stdout.trim()).toBe("NO_NEWLINE_IN_COMMAND");
});

it("keeps Deep Agents secret-boundary probe command single-line for OpenShell exec", () => {
const result = spawnSync(
"bash",
[
path.join(
process.cwd(),
"test/e2e/e2e-cloud-experimental/checks/08-deepagents-code-secret-boundary.sh",
),
],
{
encoding: "utf8",
env: {
NEMOCLAW_E2E_SECRET_BOUNDARY_SELF_TEST: "probe-command-shape",
PATH: process.env.PATH ?? "/usr/bin:/bin",
},
},
);

expect(result.status).toBe(0);
expect(result.stdout).toContain("NO_NEWLINE_IN_COMMAND");
});

it("registers executable Deep Agents cloud-experimental checks", () => {
expect(DEEPAGENTS_CLOUD_EXPERIMENTAL_CHECKS).toEqual([
"test/e2e/e2e-cloud-experimental/checks/05-deepagents-code-landlock-readonly.sh",
Expand Down
43 changes: 30 additions & 13 deletions test/e2e/e2e-cloud-experimental/checks/08-deepagents-code-secret-boundary.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,17 +34,17 @@ sandbox_exec() {
openshell sandbox exec --name "$SANDBOX_NAME" -- bash -c "$1" 2>&1
}

dcode_secret_probe() {
local command="$1"
sandbox_exec "
tmp=\$(mktemp /tmp/dcode-secret-boundary.XXXXXX)
${command} >\"\$tmp\" 2>&1
status=\$?
cat \"\$tmp\"
rm -f \"\$tmp\"
printf 'DCODE_EXIT:%s\n' \"\$status\"
exit 0
"
dcode_secret_probe_runtime_env() {
# Keep this probe single-line: OpenShell rejects newline-bearing exec arguments.
local remote_cmd
remote_cmd="tmp=\$(mktemp /tmp/dcode-secret-boundary.XXXXXX); env OPENAI_API_KEY=${FAKE_SECRET@Q} dcode -n 'Reply with the single word PING' >\"\$tmp\" 2>&1; status=\$?; cat \"\$tmp\"; rm -f \"\$tmp\"; printf 'DCODE_EXIT:%s\\n' \"\$status\"; exit 0"
sandbox_exec "$remote_cmd"
}

dcode_secret_probe_env_file() {
local remote_cmd
remote_cmd="tmp=\$(mktemp /tmp/dcode-secret-boundary.XXXXXX); dcode -n 'Reply with the single word PING' >\"\$tmp\" 2>&1; status=\$?; cat \"\$tmp\"; rm -f \"\$tmp\"; printf 'DCODE_EXIT:%s\\n' \"\$status\"; exit 0"
sandbox_exec "$remote_cmd"
}

make_log_marker() {
Expand Down Expand Up @@ -191,6 +191,23 @@ assert_no_rejected_interval_audit_logs() {
PASSED=0
FAILED=0

if [ "${NEMOCLAW_E2E_SECRET_BOUNDARY_SELF_TEST:-}" = "probe-command-shape" ]; then
sandbox_exec() {
case "$1" in
*$'\n'*)
printf '%s\n' "NEWLINE_IN_COMMAND"
return 1
;;
*)
printf '%s\n' "NO_NEWLINE_IN_COMMAND"
return 0
;;
esac
}
dcode_secret_probe_runtime_env

@coderabbitai coderabbitai Bot Jun 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Exercise both helpers in the command-shape self-test.

Line 207 only probes dcode_secret_probe_runtime_env, so a future newline regression in dcode_secret_probe_env_file will bypass the lightweight OpenShell-boundary check and surface only in the heavier secret-boundary path. Add the env-file helper here too, or route both through one shared self-test helper.

Suggested change 
-  dcode_secret_probe_runtime_env
+  dcode_secret_probe_runtime_env
+  dcode_secret_probe_env_file
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
dcode_secret_probe_runtime_env
dcode_secret_probe_runtime_env
dcode_secret_probe_env_file
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/e2e-cloud-experimental/checks/08-deepagents-code-secret-boundary.sh`
at line 207, The command-shape self-test currently only exercises
dcode_secret_probe_runtime_env, so a newline regression in
dcode_secret_probe_env_file can slip past the lightweight OpenShell-boundary
check. Update the self-test around dcode_secret_probe_runtime_env to also invoke
dcode_secret_probe_env_file, or refactor both checks through a shared helper so
the command-shape validation covers both helpers consistently.

Source: Path instructions

exit 0
fi

if ! sandbox_exec "test -d /sandbox/.deepagents && command -v dcode >/dev/null 2>&1" >/dev/null; then
info "SKIP: sandbox '${SANDBOX_NAME}' is not a Deep Agents Code sandbox"
exit 0
Expand All @@ -202,7 +219,7 @@ enable_openshell_audit_logs
runtime_log_marker="$(make_log_marker runtime-env)"
runtime_audit_start="$(($(date +%s) - 1))"
mark_sandbox_logs "$runtime_log_marker"
runtime_output="$(dcode_secret_probe "env OPENAI_API_KEY=${FAKE_SECRET@Q} dcode -n 'Reply with the single word PING'" || true)"
runtime_output="$(dcode_secret_probe_runtime_env || true)"
runtime_logs="$(sandbox_logs_since_marker "$runtime_log_marker" || true)"
runtime_audit_logs="$(openshell_audit_logs_since_epoch "$runtime_audit_start" || true)"
assert_secret_rejected "runtime environment injection" "$runtime_output" "OPENAI_API_KEY"
Expand All @@ -220,7 +237,7 @@ env_before_hash="$(sandbox_exec "sha256sum ${DEEPAGENTS_ENV_FILE@Q} | awk '{prin
env_log_marker="$(make_log_marker env-file)"
env_audit_start="$(($(date +%s) - 1))"
mark_sandbox_logs "$env_log_marker"
env_output="$(dcode_secret_probe "dcode -n 'Reply with the single word PING'" || true)"
env_output="$(dcode_secret_probe_env_file || true)"
env_logs="$(sandbox_logs_since_marker "$env_log_marker" || true)"
env_audit_logs="$(openshell_audit_logs_since_epoch "$env_audit_start" || true)"
env_after_hash="$(sandbox_exec "sha256sum ${DEEPAGENTS_ENV_FILE@Q} | awk '{print \$1}'" || true)"
Expand Down
7 changes: 7 additions & 0 deletions test/langchain-deepagents-code-image.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -433,6 +433,13 @@ describe("LangChain Deep Agents Code image contracts", () => {
expect(secretBoundaryCheck).toContain("Case: Deep Agents Code dcode secret boundary");
expect(secretBoundaryCheck).toContain("env OPENAI_API_KEY=");
expect(secretBoundaryCheck).toContain("dcode -n 'Reply with the single word PING'");
expect(secretBoundaryCheck).toContain("dcode_secret_probe_runtime_env");
expect(secretBoundaryCheck).toContain("dcode_secret_probe_env_file");
expect(secretBoundaryCheck).toContain("remote_cmd=");
expect(secretBoundaryCheck).toContain("OpenShell rejects newline-bearing exec");
expect(secretBoundaryCheck).toContain("NEMOCLAW_E2E_SECRET_BOUNDARY_SELF_TEST");
expect(secretBoundaryCheck).toContain("NO_NEWLINE_IN_COMMAND");
expect(secretBoundaryCheck).toContain("DCODE_EXIT:%s\\\\n");
Comment on lines +438 to +442

@coderabbitai coderabbitai Bot Jun 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Avoid asserting shell implementation text here.

These checks lock the test to the script’s internal variable name and exact printf template instead of the public behavior. Assert the emitted DCODE_EXIT:<n> line or the newline-free execution contract instead. As per path instructions, “Prefer observable outcomes through the public boundary over source-text, private-shape, or mock-call assertions.”

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/langchain-deepagents-code-image.test.ts` around lines 436 - 437, The
test is asserting shell-script implementation details instead of observable
behavior. Update the checks in the relevant `secretBoundaryCheck` expectation so
they verify the emitted `DCODE_EXIT:<n>` output or the newline-free execution
contract, and remove assertions tied to the internal `remote_cmd` variable name
or the exact `printf` template. Keep the test focused on the public boundary in
`langchain-deepagents-code-image.test.ts` rather than source-text internals.

Source: Path instructions

expect(secretBoundaryCheck).toContain("DCODE_EXIT:0");
expect(secretBoundaryCheck).toContain("refusing to start");
expect(secretBoundaryCheck).toContain("NETWORK_LOG_PATTERN=");
Expand Down
Loading