Skip to content

fix(desktop): stop double-quoting expandRemotePath fragments in the SSH spawn path - #96187

Closed
koltyj wants to merge 2 commits into
NousResearch:mainfrom
koltyj:fix/ssh-spawn-double-quoting
Closed

koltyj wants to merge 2 commits into
NousResearch:mainfrom
koltyj:fix/ssh-spawn-double-quoting

Conversation

@koltyj

@koltyj koltyj commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Symptom

Since the managed SSH remote update engine landed, every Desktop SSH backend spawn on a POSIX remote hangs ~20s and dies, in a loop: the Bots view shows "Could not check 's Bot Chat registry (Timed out connecting to profile …)", and each retry strands an orphaned flock-queue python on the remote. A telltale artifact is a directory literally named ' in the remote $HOME, containing a quote-mangled copy of the .hermes-update-in-progress.mutex path.

Root cause

expandRemotePath() returns an already-quoted shell fragment ("$HOME"'/path'), meant for direct interpolation into the remote command string. Three call sites wrapped that fragment in shq() a second time:

  1. withRemoteUpdateMutexshq(mutexPath) in the python3 -c argv
  2. buildSpawnCommand — the reservation=/lock=/owner_file= payload assignments
  3. buildOwnedStaleTerminationCommand — the identity values (path=/home=/token=)

The remote shell strips only one layer, so python receives a mutex path with literal quote characters (→ the ' shadow directory), the payload's while ! mkdir "$reservation" spins forever on a literal "$HOME"'…' path (→ the 20s hang + orphan queue), and the stale-owner identity check can never match live argv (→ cleanup always REFUSED, so the orphans accumulate).

Fix

Interpolate the fragments raw at those three sites, exactly as the neighboring mkdir -p "$(dirname …)"/rm -f call sites already do. No behavior change beyond correct quoting.

Regression test

The new test parses the composed spawn command with a real sh (python3 shimmed on PATH to capture argv) — the same single parse the remote login shell performs — then evaluates the payload's assignment prefix and requires the mutex path and reservation/lock/owner paths to come out fully expanded. It fails on the previous code with:

AssertionError: mutex path must reach python fully expanded, with no quote characters

and passes with the fix.

…SH spawn path

expandRemotePath() returns an already-quoted shell fragment
("$HOME"'/path'), but three call sites wrapped its output in shq()
again: the withRemoteUpdateMutex python argv, the reservation/lock/
owner_file assignments in buildSpawnCommand, and the identity values in
buildOwnedStaleTerminationCommand.

The remote shell strips only one quoting layer, so python received a
mutex path with literal quote characters in it (creating a directory
literally named ' in $HOME), and the payload's mkdir "$reservation"
loop spun on a path that can never exist. Every Desktop SSH backend
spawn hung until the connect timeout, retried, and left an orphaned
flock queue behind; stale-owner cleanup always printed REFUSED for the
same reason.

The regression test parses the composed command with a real sh — the
same parse the remote login shell performs — and requires the mutex
path and the payload's reservation paths to come out fully expanded.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/desktop Electron desktop app (apps/desktop/*) backend/ssh SSH remote execution duplicate This issue or pull request already exists labels Aug 27, 2026
@alt-glitch

Copy link
Copy Markdown
Contributor

This was generated by AI during triage.

Duplicate of #96084. Both repair the same Desktop SSH remote-lifecycle double-quoting mechanism; #96084 is the earlier core-team salvage with the broader companion POSIX-spawn repair.

…and stale-identity sites independently; this branch now carries only the remaining withRemoteUpdateMutex shq(mutexPath) fix and the real-sh-parse regression test
@koltyj

koltyj commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Rebased via merge after beb212d landed — that commit independently fixed two of the three double-quoting sites this PR covered (the payload's reservation/lock/owner_file assignments and the stale-termination identity values). The remaining delta here is the third site: withRemoteUpdateMutex still passes shq(mutexPath) into the python3 -c argv, so the updater mutex flocks a literal-quoted filename (and recreates the ' shadow directory in $HOME) instead of the real .hermes-update-in-progress.mutex sidecar — the update/spawn race the mutex exists to close is still open. The regression test in this PR fails on current main for exactly that path and covers the two already-fixed sites as well.

@SZWzz

SZWzz commented Sep 4, 2026

Copy link
Copy Markdown

Heads-up on overlap: this fixes the same withRemoteUpdateMutex double-quoting site as #96260 (and #96187).

Whichever one lands fixes the quoting fault, so no argument from me either way — but for whoever reviews: #96260 covers this same one-line mutex change plus two adjacent correctness defects in the same spawn chain that the one-line fix alone leaves behind:

  • the published lockfile pid is a JSON string ("__PID__" substituted unquoted), while readLockfile requires an integer and the reuse regex only matches "pid":<digits> — so every fresh spawn's ownership record is malformed;
  • the detached setsid/nohup shell echoed $! from both the inner and outer shells, so $child captured a two-line value and the sed substitution was invalid under POSIX sh (now guarded: non-integer $child fails closed with exit 76).

It also pins the contract with argv-level regression tests (asserting the exact argv the remote python receives, under dash), which is the gap that let this site regress silently in the first place — the existing cmd.includes(...) assertions pass under both the broken and fixed forms.

Happy to rebase/adjust if a maintainer prefers to land a minimal one first.

@alt-glitch alt-glitch added area/install-update Installer, updater, packaging, wheels, doctor sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades and removed duplicate This issue or pull request already exists labels Sep 4, 2026
@alt-glitch

Copy link
Copy Markdown
Contributor

This was generated by AI during triage.

Re-triaged after the rebase: no longer marking this a duplicate of #96084. The remaining withRemoteUpdateMutex double-quoting site is still on main; this PR now competes with #99189 (same one-line fix) and is a subset of #96260 (mutex fix + integer lockfile pid + $! capture). Leaving all three open for a maintainer to pick.

@kshitijk4poor

Copy link
Copy Markdown
Contributor

Salvaged into #104206 — your commit 93ff480d83 is cherry-picked there with authorship preserved (author Kolton Jacobs, committer me), so credit lands on main under your name; contributors/emails mapping added for @koltyj.

Thread points, addressed:

  • alt-glitch's original "duplicate of fix(desktop): fresh managed-SSH spawns no longer wedge on dash or double-quoted paths (salvage #96061) #96084" is stale, as the re-triage noted: fix(desktop): fresh managed-SSH spawns no longer wedge on dash or double-quoted paths (salvage #96061) #96084 fixed the reservation/lock/owner_file and stale-termination sites; withRemoteUpdateMutex was the one remaining shq(expandRemotePath(...)) site and is still live on origin/main (verified by running the composed command through a real sh with a python3 shim: sys.argv[1] arrives as '/home/hermes/.hermes/.hermes-update-in-progress.mutex' with the quotes).
  • Your real-sh-parse test was chosen over the two competing regex-assertion tests precisely because it proves the expansion instead of matching the string. Small follow-up commit on top: win32 skip like the sibling shell-out tests, sentinel guard before the i=0; slice so a renamed loop marker can never execute the real payload locally, single temp root, quoted redirect, reuse of the file's exec.
  • Rebase-via-merge commit 24cae37f0b dropped (only the substantive commit is carried).

The competing #99189 / #96260 fix the same line; #96260's separate lockfile-pid concern stays open there. Auto-merge (rebase) is armed. Thanks for finding it first.

kshitijk4poor added a commit that referenced this pull request Sep 6, 2026
Attribution mapping so the cherry-picked #96187 commit resolves to its
GitHub account (@jonpol01 is already in the legacy map).
kshitijk4poor pushed a commit that referenced this pull request Sep 6, 2026
`apps/desktop/'` is not a real path. It is a literal single-quote directory
holding a full absolute path as nested subdirectories:

    apps/desktop/'/var/folders/5h/.../hermes-update-mutex-LMF9y5/home/.hermes-update-in-progress.mutex'

Both files are 0 bytes, nothing in the tree references them, and the leading
and trailing `'` are part of the filenames. They are the fingerprint of the
double-quoted mutex path in `withRemoteUpdateMutex()`: the path reaches Python
with its shell quotes still attached, so it is treated as CWD-relative and
`os.makedirs()` materialises the whole absolute path under whatever directory
the process happened to be in. Running
`apps/desktop/electron/remote-lifecycle.test.ts` from `apps/desktop`
reproduces it on the spot.

They were swept in by a `git add .` in 3662057, an unrelated menu-label
commit.

This only removes the committed artifact. The generator is a separate concern
already covered by open PRs (#99189, #96187, #96260) and issues (#99133,
#96212, #96188); those repair the quoting but none of them deletes these two
files, so the litter would survive whichever one lands.
kshitijk4poor added a commit that referenced this pull request Sep 6, 2026
Review follow-ups on the #96187 cherry-pick: skip on win32 like the
sibling tests that shell out; reuse the file's `exec` helper; one temp
root so a failing second mkdtemp cannot leak the shim dir; quote the
shim's redirect target; guard the payload-prefix sentinel so a renamed
loop marker can never make the test execute the real spawn payload;
drop the lockMetadata fields the assertions never read. Move the
mutexPath contract into withRemoteUpdateMutex's doc comment instead of
a third inline restatement.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/install-update Installer, updater, packaging, wheels, doctor backend/ssh SSH remote execution comp/desktop Electron desktop app (apps/desktop/*) P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants