Skip to content

fix(tunnel): verify process identity before stopping cloudflared - #7644

Merged
jyaunches merged 12 commits into
mainfrom
fix/cloudflared-stop-verify-pid-identity
Jul 31, 2026
Merged

fix(tunnel): verify process identity before stopping cloudflared#7644
jyaunches merged 12 commits into
mainfrom
fix/cloudflared-stop-verify-pid-identity

Conversation

@latenighthackathon

@latenighthackathon latenighthackathon commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

stopService signalled whatever PID was recorded in cloudflared.pid after only an isAlive check. If the recorded cloudflared had exited and the OS recycled its PID to an unrelated process, that process could receive SIGTERM or SIGKILL. This verifies the live PID still names cloudflared before either signal and removes a stale PID file without signalling a bystander.

Related Issue

Closes #7643

Changes

  • Add an injectable process-control seam so PID reuse and signalling are tested deterministically without spawning or signalling host processes.
  • Revalidate the recorded PID's cloudflared identity before SIGTERM.
  • Revalidate identity again immediately before SIGKILL escalation.
  • Remove the stale PID file without signalling when the PID belongs to another process.
  • Add regression coverage for both an already-recycled PID and recycling between SIGTERM and escalation.
  • Add deterministic coverage for persistent cloudflared escalation (SIGTERMSIGKILL) and PID-file cleanup.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: Internal service-lifecycle safety behavior; commands, configuration, and supported platforms are unchanged.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Codex Desktop completed a serialized nine-category review of exact signed/Verified head 5700f6a22d2dd791b63bba7ddd7cf460800f1299 on current base 94c0b07efb7bcc5d9ddd1fd590a2da26514bbcec. Categories 1–8 PASS. Category 9 has a non-blocking residual warning: command-line identity lookup and POSIX signalling are not one OS-atomic process-handle operation. This is the explicit conservative compatibility boundary accepted by cloudflared stop can SIGKILL an unrelated process when the recorded PID has been recycled #7643; the exact Advisor reports no blockers, CodeRabbit’s actionable threads are resolved, and the deterministic tests cover a pre-signal bystander plus recycling before escalation. Exact diff fingerprint: 44af53152b9cd4b61a10557334ac4bea0a101a6288f19c932e24eef295d450bc (git diff --binary origin/main...HEAD | shasum -a 256).
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: Exact head 5700f6a22d2dd791b63bba7ddd7cf460800f1299 changes internal tunnel process-lifecycle safety, regression coverage, and a typed readiness test fixture. User-facing commands, configuration, supported platforms, and documented tunnel behavior are unchanged. Focused tunnel tests pass 38/38, the readiness fixture passes 2/2, and CLI build/typecheck plus normal commit/pre-push hooks pass.
  • Agent: Codex Desktop (serialized main task)

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit: Not applicable; the DGX Station test-only fixture update carries the required immutable source-revision field and does not change Station runtime behavior.
  • Station profile/scenario: Not applicable.
  • Result: Not applicable.
  • Supporting evidence: Not applicable.

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed
  • Targeted behavior tests pass for the current change set — tunnel service tests 38/38; Station release-marker tests 2/2; CLI build and typecheck pass at exact head 5700f6a22.
  • Applicable broad gate passed — fresh exact-head GitHub CI/E2E/automated review is running after the test-only repair.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: latenighthackathon latenighthackathon@users.noreply.github.com

Summary by CodeRabbit

  • Bug Fixes
    • Improved tunnel shutdown safety by verifying the recorded process is the expected service before sending termination signals.
    • Prevented unrelated processes from being stopped when a process ID has been recycled.
    • Stale process ID records are now cleaned up automatically, including scenarios where the PID’s liveness changes during shutdown.
    • Ensured escalation during shutdown does not go beyond the intended first termination signal unless the process still matches expectations.

stopService signalled whatever PID was recorded in cloudflared.pid after
only an isAlive check. If the recorded cloudflared had exited and the OS
recycled its PID to an unrelated process, that process was SIGTERM/
SIGKILLed. cloudflared is spawned detached and can exit early (for
example on an invalid tunnel token), leaving a stale pid file, so PID
reuse is a realistic path, and under an elevated topology the recycled
PID can be a system process. Every other kill site in this module
already gates on a command-line identity match; stopService was the lone
unverified killer.

Verify the live PID still names cloudflared (the same readProcessCommandLine
plus commandLineNamesCloudflared check readCloudflaredState uses) before
signalling, and drop the stale pid file when it does not. A null or
unreadable cmdline stays conservative and proceeds, matching
readCloudflaredState.

Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Changes

PID termination safety

Layer / File(s) Summary
Add process-control seam and shutdown identity guards
src/lib/tunnel/services.ts
stopService validates that the recorded PID belongs to cloudflared before signaling, rechecks identity before escalation, and supports injected process-control operations through stopAll.
Test recycled PID handling
src/lib/tunnel/services.test.ts
Scripted process-control tests verify unrelated processes receive no signals and recycled PIDs do not receive SIGKILL while stale PID files are removed.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: cv, thabhelo, brandonpelfrey

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes match #7643 by verifying cloudflared identity before signaling, removing stale PID files, and rechecking before SIGKILL.
Out of Scope Changes check ✅ Passed The diff stays focused on the stop-path fix and its regression tests, with no obvious unrelated changes.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: verifying process identity before stopping cloudflared.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/cloudflared-stop-verify-pid-identity

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 5700f6a in the fix/cloudflared-stop... branch remains at 96%, unchanged from commit 94c0b07 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 5700f6a in the fix/cloudflared-stop... branch remains at 81%, unchanged from commit 94c0b07 in the main branch.

Show a code coverage summary of the most impacted files.
File main 94c0b07 fix/cloudflared-stop... 5700f6a +/-
src/lib/shields/index.ts 69% 69% 0%
src/lib/sandbox...rce-identity.ts 88% 88% 0%
src/lib/tunnel/services.ts 73% 75% +2%
src/lib/domain/.../connect-env.ts 89% 97% +8%

Updated July 31, 2026 16:01 UTC

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/tunnel/services.test.ts`:
- Around line 494-513: Update the test around stopAll, the /proc state
inspection, and its assertions to use try/finally cleanup. Ensure
logSpy.mockRestore() and the bystander process termination via
process.kill(bpid, "SIGKILL") run in finally blocks even when stopAll,
inspection, or assertions throw, while preserving the existing process-state
checks.
- Around line 482-517: Replace the real sleep process, /proc inspection, and
process.kill cleanup in the test “does not signal a recycled PID whose process
is not cloudflared” with the tunnel adapter/fake pattern used by the surrounding
tests. Configure the fake process lookup to model a recycled PID belonging to a
non-cloudflared process, assert that stopAll does not signal it, and retain the
PID-file removal assertion without broad host interactions.

In `@src/lib/tunnel/services.ts`:
- Around line 352-361: Revalidate PID ownership immediately before the SIGKILL
escalation in the tunnel shutdown flow, using readProcessCommandLine and
commandLineNamesCloudflared as in the existing pre-SIGTERM check. If the live
PID no longer identifies cloudflared, log that it is not running, remove the
stale PID via removePid, and return without sending SIGKILL.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e0f44edf-f2ab-4f39-beaf-8f8056a51fda

📥 Commits

Reviewing files that changed from the base of the PR and between 1bc0283 and 2ae67fb.

📒 Files selected for processing (2)
  • src/lib/tunnel/services.test.ts
  • src/lib/tunnel/services.ts

Comment thread src/lib/tunnel/services.test.ts Outdated
Comment thread src/lib/tunnel/services.test.ts
Comment thread src/lib/tunnel/services.ts Outdated
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: None

1 optional E2E recommendation
  • tunnel-lifecycle

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

…rocess ops in tests

Address review feedback on the recycled-PID stop fix.

Re-run the command-line identity check immediately before the SIGKILL
escalation, not only before SIGTERM: cloudflared can exit and have its
PID recycled during the up-to-3s exit poll, so the escalation could
otherwise kill an unrelated process.

Route stopService process operations (isAlive / commandLine / signal)
through an injectable ProcessControl. The regression tests now drive a
deterministic fake that models PID reuse instead of spawning a real
process and reading /proc/<pid>/stat, which was not portable to the
macOS Vitest runner and did not follow the tunnel adapter/fake test
convention. Adds coverage for both the recycled-before-SIGTERM and
recycled-during-poll paths.

Signed-off-by: latenighthackathon <latenighthackathon@users.noreply.github.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/lib/tunnel/services.ts (1)

146-162: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider consolidating the duplicated identity-check logic.

pidIsOurs (new) and the inline check in readCloudflaredState (Line 283: cmdline !== null && !commandLineNamesCloudflared(cmdline)) both implement the same "null-or-cloudflared" rule, but readCloudflaredState still calls readProcessCommandLine/process.kill directly instead of going through ProcessControl. Extracting a shared predicate (or having readCloudflaredState accept an injectable ProcessControl) would avoid the duplication and let that function benefit from the same deterministic test seam.

As per path instructions (src/lib/README.md), "align with the adapter/injection pattern where process/FS interactions are involved."

Also applies to: 282-285, 364-367

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/tunnel/services.ts` around lines 146 - 162, Consolidate the
duplicated PID identity check by reusing pidIsOurs from readCloudflaredState and
related lifecycle paths. Route command-line lookup and signaling through the
injected ProcessControl interface instead of calling readProcessCommandLine or
process.kill directly, preserving the null-or-cloudflared rule and enabling
deterministic tests.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/tunnel/services.ts`:
- Around line 146-162: Consolidate the duplicated PID identity check by reusing
pidIsOurs from readCloudflaredState and related lifecycle paths. Route
command-line lookup and signaling through the injected ProcessControl interface
instead of calling readProcessCommandLine or process.kill directly, preserving
the null-or-cloudflared rule and enabling deterministic tests.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 160ce322-dbab-4ff4-8209-8113a41afcaa

📥 Commits

Reviewing files that changed from the base of the PR and between 2ae67fb and 5468a3b.

📒 Files selected for processing (2)
  • src/lib/tunnel/services.test.ts
  • src/lib/tunnel/services.ts

@cv cv added the v0.0.97 label Jul 27, 2026
@apurvvkumaria apurvvkumaria self-assigned this Jul 27, 2026
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@cjagwani cjagwani added v0.0.98 and removed v0.0.97 labels Jul 28, 2026
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@wscurran wscurran added area: networking DNS, proxy, TLS, ports, host aliases, or connectivity bug-fix PR fixes a bug or regression labels Jul 28, 2026
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@cjagwani

Copy link
Copy Markdown
Collaborator

I reviewed the current exact two-file diff. The identity checks are scoped to accepted issue #7643, and the fail-open behavior for an unreadable command line matches that issue’s explicit compatibility boundary. I’m keeping this as a comment, not a changes-requested review.

Two mechanical items remain before exact-head approval:

  • the PR base is still a4a5aa2d8, while current main is 795de982e; maintainer edits are disabled, so please merge current main into this branch;
  • please add the advisor-identified scripted ProcessControl case where PID 4242 remains live and names cloudflared through escalation, asserting SIGTERM then SIGKILL and PID-file removal. The rejection paths are covered, but this changed allowed branch is not.

Once that quiet refreshed head is available, I’ll babysit its focused test, current CI, signatures, and final approval.

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head approval for f350d4b after the one-hour quiet check. All 50 GitHub checks are green; the focused tunnel suite passes 37/37 locally; CLI typecheck passes with the PR-pinned dependency tree; all seven commits are Verified; and there are no unresolved major findings. Security review PASS across credentials, input handling, authorization, dependencies, error exposure, data protection, configuration, tests, and system/process safety. The stop path now validates cloudflared identity before SIGTERM and revalidates before SIGKILL, with deterministic recycled-PID regressions. The conservative unreadable-command-line behavior is the explicit accepted #7643 contract. GitHub reports no conflict, and the user waiver applies to base-only refreshes.

apurvvkumaria and others added 5 commits July 31, 2026 04:22
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@jyaunches
jyaunches merged commit 57f73a5 into main Jul 31, 2026
74 of 75 checks passed
@jyaunches
jyaunches deleted the fix/cloudflared-stop-verify-pid-identity branch July 31, 2026 16:17
@senthilr-nv senthilr-nv mentioned this pull request Aug 1, 2026
23 tasks
senthilr-nv added a commit that referenced this pull request Aug 1, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Adds the canonical dated changelog entry for `v0.0.100` so the
maintainer release plan can verify the pre-tag documentation
prerequisite. The entry summarizes the user-facing changes merged since
`v0.0.99` and links to the relevant guides.

## Changes

- Add `docs/changelog/2026-07-31.mdx` with the exact `## v0.0.100`
heading.
- Cover restored OpenClaw pairing, transactional replacement, Deep
Agents Code, onboarding recovery, lifecycle cleanup, Hermes builds, host
provenance, documentation, and trusted E2E evidence.
- Distinguish active Docker and Kubernetes runtime-bundle enforcement
from the still-inactive managed shared-state transaction foundation.

## Source Coverage

The release entry maps the doc-impacting merged PRs in the
`v0.0.99..main` release range to `docs/changelog/2026-07-31.mdx`: #8021,
#8024, #7973, #8028, #7947, #7788, #7884, #8023, #7969, #8020, #7989,
#8000, #7907, #7942, #7567, #8013, #7955, #8017, #8014, #8015, #7629,
#7644, #7821, #7971, and #7991.

PR #7974 was reviewed after the final rebase and excluded because it
changes internal maintainer-skill attribution policy and tests only; it
does not change a user-facing product or documentation surface.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification: the
changelog contract test validates the dated entry, version heading, SPDX
form, and route constraints.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `docs/changelog/2026-07-31.mdx`; exact-head review passed
for `6093f44f`; writing rules and documentation style reviewed; `npx
vitest run test/changelog-docs.test.ts` passed 6/6; `npm run docs`
passed with zero Fern errors and two generic Fern upgrade notices.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 6093f44 -->
<!-- docs-review-agents-blob-sha: 3dd7c24 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable; no DGX Station host script changed.
- Station profile/scenario: Not applicable.
- Result: Not applicable.
- Supporting evidence: Not applicable.

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: `npx
vitest run test/changelog-docs.test.ts` passed 6/6 at `6093f44f`.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable to a dated
prose-only release entry.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
validation passed with zero errors; Fern emitted two generic upgrade
notices.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— the changelog entry has the required parser-safe MDX SPDX header;
dated changelog entries intentionally do not use page frontmatter.

---

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
  * Added release notes for v0.0.100.
* Documented improvements to restore pairing, sandbox replacement,
onboarding recovery, lifecycle cleanup, runtime handling, build support,
host readiness, and end-to-end validation.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: networking DNS, proxy, TLS, ports, host aliases, or connectivity bug-fix PR fixes a bug or regression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cloudflared stop can SIGKILL an unrelated process when the recorded PID has been recycled

7 participants