fix(tunnel): detect gateway with rewritten bare 'openclaw' argv (#4951) by abhi-0906 · Pull Request #4960 · NVIDIA/NemoClaw

abhi-0906 · 2026-06-08T13:36:56Z

Summary

nemoclaw tunnel stop (and its alias nemoclaw stop) reported success and exited 0 while the in-sandbox OpenClaw gateway — and its channel pollers — kept running. The stop script's process matcher never found the gateway because OpenClaw rewrites its own argv to a bare openclaw after startup. This teaches the matcher that third argv form so the gateway is actually detected and stopped.

Related Issue

Fixes #4951

Root cause

stopSandboxChannels runs an in-sandbox GATEWAY_STOP_SCRIPT (src/lib/tunnel/services.ts) whose find_gateway_pids scans ps -eo args= and matches the gateway by argv. It recognized only:

openclaw-gateway (the re-execed binary name), and
openclaw gateway run … (the launcher command nemoclaw-start runs).

But OpenClaw sets process.title = 'openclaw' after startup, so the live gateway's argv is just openclaw with no gateway suffix. As a result find_gateway_pids returned empty → the script exit 1 → reportStopResult interprets exit 1 as "gateway was not running" → the command prints success and exits 0, leaving the gateway and Slack/Telegram/Discord pollers alive. (Same process.title root cause as the sandbox HEALTHCHECK bug, NVB#6282411 / NVB#6282413.)

Changes

src/lib/tunnel/services.ts — add a third argv form to the find_gateway_pids awk matcher: a bare openclaw anchored to end-of-string, so it still rejects unrelated names like openclawish. Document the three forms and why example argv tokens must stay out of the awk program body (awk's own argv is captured by the concurrent ps snapshot, so an in-program literal like openclaw gateway makes awk match itself and the kill/verify scan never drains). Export GATEWAY_STOP_SCRIPT for end-to-end testing.
src/lib/tunnel/services-sandbox.test.ts — assert the matcher contains all three argv forms, and add two Linux-gated tests that execute the real GATEWAY_STOP_SCRIPT against live processes reproducing each argv form: the bare-argv gateway is found and killed (exit 0), and a non-gateway decoy is spared (exit 1).

Type of Change

Code change (feature, bug fix, or refactor)
Code change with doc updates
Doc only (prose changes, no code sample modifications)
Doc only (includes code sample changes)

Verification

npx prek run --all-files passes
npm test passes
Tests added or updated for new or changed behavior
No secrets, API keys, or credentials committed
Docs updated for user-facing behavior changes
npm run docs builds without warnings (doc changes only)
Doc pages follow the style guide (doc changes only)
New doc pages include SPDX header and frontmatter (new pages only)

Ran locally on the affected files: biome check (clean), tsc -p jsconfig.json (clean), vitest run src/lib/tunnel/services-sandbox.test.ts (17 passed, 2 Linux-gated execution tests skipped on the Windows dev box), plus source-shape and test-size checks. The two Linux-gated execution tests were validated against the compiled script on Ubuntu 24.04 (WSL): bare openclaw killed → exit 0; decoy spared → exit 1.

Summary by CodeRabbit

Bug Fixes
- Improved in-sandbox “tunnel stop” to reliably terminate all relevant gateway process naming variants, including the “bare” gateway form, while sparing unrelated processes.
- Strengthened the container gateway health-check to safely validate PID reuse by parsing PID/start-time data and verifying the process matches the expected gateway command name.
Tests
- Added Linux-only end-to-end coverage for stop-script matching across multiple gateway argv patterns, including ensuring decoy processes are not terminated.
- Added a regression test for the health-check fallback when the recorded PID start time is stale.

Signed-off-by: Abhimanyu Kumar abhimanyukumar7290@gmail.com

Signed-off-by: Julie Yaunches jyaunches@nvidia.com

copy-pr-bot · 2026-06-08T13:37:01Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

coderabbitai · 2026-06-08T13:37:11Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR updates gateway stop matching for the bare openclaw process form and changes PID file writing, reading, and health checks to support an optional recorded start time.

Changes

Gateway stop and PID tracking

Layer / File(s)	Summary
Stop script matcher and tests `src/lib/tunnel/services.ts`, `src/lib/tunnel/services-sandbox.test.ts`	The stop script documentation and awk matcher include the bare `openclaw` process form, and sandbox tests assert the updated matching behavior with unit and Linux-only executed coverage.
PID file write and read `scripts/nemoclaw-start.sh`, `Dockerfile`, `test/nemoclaw-start-gateway-health.test.ts`, `test/sandbox-provisioning.test.ts`	`record_gateway_pid` writes `pid` or `pid starttime`, the watchdog and healthcheck read only the first PID field and validate start-time reuse, and health tests accept the extended pidfile format and stale-start detection.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

area: cli, area: sandbox

Suggested reviewers

cv
jyaunches
ericksoa

Poem

🐰 I hop where the gateway names appear,
openclaw plain and clear.
PID files whisper, “read me light,”
one token first, then all is right.
Stop scripts nibble, tests take flight.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main fix: tunnel stop now detects gateways whose argv is rewritten to bare 'openclaw'.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

wscurran · 2026-06-08T14:44:27Z

✨ Thanks for submitting this detailed PR about detecting the gateway with rewritten bare 'openclaw' argv. This proposes a way to fix the regression in the tunnel stop functionality.

Related open issues:

#4951 [All Platforms][CLI&UX] nemoclaw tunnel stop reports gateway not running and exits 0 while in-sandbox openclaw gateway keeps running

wscurran · 2026-06-08T14:44:27Z

✨ Thanks for submitting this detailed PR about detecting the gateway with rewritten bare 'openclaw' argv. This proposes a way to fix the regression in the tunnel stop functionality.

Related open issues:

#4951 [All Platforms][CLI&UX] nemoclaw tunnel stop reports gateway not running and exits 0 while in-sandbox openclaw gateway keeps running

prekshivyas

Correct. `process.title = 'openclaw'` after startup made `find_gateway_pids` return empty, so `tunnel stop` silently exited 0. The new awk arm `/(^|[[:space:]/])openclaw[[:space:]]*$/` anchors at EOL, avoiding false matches on `openclaw-gateway` or `openclawish`. The live-process tests with `exec -a 'openclaw' cat` are the right guard for this.

Minor: the 300 ms post-SIGTERM sleep before the liveness check could flake on a heavily loaded CI runner. A short poll loop would be more robust — not a blocker.

…IA#4951) `nemoclaw tunnel stop` could not stop the in-sandbox gateway: OpenClaw rewrites its own argv via process.title after startup, so the running gateway shows just `openclaw` with no `gateway` suffix. The awk matcher in GATEWAY_STOP_SCRIPT only recognized `openclaw-gateway` and `openclaw gateway`, so find_gateway_pids returned empty, reportStopResult misread exit 1 as "not running", and the command exited 0 while the gateway (and its channel pollers) kept running. Add a third argv form to the matcher: a bare `openclaw` anchored to end-of-string (so it still rejects names like `openclawish`). The example argv tokens are kept out of the awk program text itself, because awk's argv is captured by the concurrent `ps` snapshot and any such literal would make awk match itself and prevent the scan from draining. Export GATEWAY_STOP_SCRIPT and add Linux-gated tests that execute it against real processes reproducing each argv form, asserting the bare gateway is killed (exit 0) and a non-gateway decoy is spared (exit 1). Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

github-actions · 2026-06-26T12:31:58Z

Vitest E2E Scenario Results — ❌ Some jobs failed

Run: 28238128513
Workflow ref: ci/pr-4960-40fda48c
Requested scenarios: (default — all supported)
Requested jobs: sandbox-survival-vitest
Summary: 0 passed, 1 failed, 0 cancelled, 0 skipped

Job	Result
sandbox-survival-vitest	❌ failure

Failed jobs: sandbox-survival-vitest. Check run artifacts for logs.

github-actions · 2026-06-26T12:32:33Z

Selective E2E Results — ✅ All requested jobs passed

Run: 28237439156
Target ref: ci/pr-4960-40fda48c
Requested jobs: cloud-onboard-e2e,tunnel-lifecycle-e2e,sandbox-survival-e2e,issue-2478-crash-loop-recovery-e2e
Summary: 4 passed, 0 failed, 0 cancelled, 0 skipped

Job	Result
cloud-onboard-e2e	✅ success
issue-2478-crash-loop-recovery-e2e	✅ success
sandbox-survival-e2e	✅ success
tunnel-lifecycle-e2e	✅ success

prekshivyas · 2026-07-05T19:51:02Z

@abhi-0906 this is still wanted — the target bug #4951 (nemoclaw tunnel stop reports gateway not running) is still open and nothing has superseded it, and the PR is approved. But the branch is ~235 commits behind main and no longer merges cleanly.

I looked at merging origin/main in: the core fix (src/lib/tunnel/services.ts) merges fine, but there are conflicts in files this PR also changed where main has since evolved the same regions — notably scripts/nemoclaw-start.sh (main reworked record_gateway_pid to capture /proc/<pid>/stat starttime + watchdog-kill records), plus Dockerfile, test/nemoclaw-start-gateway-health.test.ts, and src/lib/tunnel/services-sandbox.test.ts. Reconciling your argv-detection changes with main's new gateway-PID recording is behavior-sensitive, so it's better done by you than guessed at on your behalf.

Could you run git merge origin/main (not rebase — the branch is published) and resolve those, then push? Once it's mergeable I can shepherd it through the merge. Thanks!

Reconcile with main's gateway-PID identity recording (NVIDIA#4952 line): the branch's interim starttime capture in record_gateway_pid, the Dockerfile HEALTHCHECK starttime hardening, and the stale-starttime provisioning test are all superseded by main's caller-supplied start-identity design and its existing coverage, so those files resolve to main verbatim. The surviving delta is the tunnel stop argv-detection fix in src/lib/tunnel/services.ts and its tests. Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>

abhi-0906 · 2026-07-07T09:06:22Z

@prekshivyas Merged origin/main in (73790af). Resolution notes:

src/lib/tunnel/services.ts and its tests merged clean; services-sandbox.test.ts now imports from ./services per main's convention.
scripts/nemoclaw-start.sh, Dockerfile, test/nemoclaw-start-gateway-health.test.ts, test/sandbox-provisioning.test.ts resolve to main verbatim: this branch's interim starttime capture is superseded by main's caller-supplied start-identity design, and the stale-starttime test duplicates main's "rejects a live PID whose start identity differs from the recorded gateway".
Net delta vs main is now 3 files: the argv-detection fix + its tests + a small dcode wrapper test cleanup. Tunnel suite passes locally; the executed GATEWAY_STOP_SCRIPT tests need the Linux runners.