Skip to content

fix(terminal): cap ssh handshake at 10s so hung sshd surfaces fast - #2434

Merged
HongmingWang-Rabbit merged 23 commits into
stagingfrom
fix/terminal-ssh-connect-timeout
May 1, 2026
Merged

fix(terminal): cap ssh handshake at 10s so hung sshd surfaces fast#2434
HongmingWang-Rabbit merged 23 commits into
stagingfrom
fix/terminal-ssh-connect-timeout

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

Summary

When a workspace EC2's sshd is unresponsive (mid-restart, SG drop, AMI without ec2-instance-connect installed), the canvas Terminal tab silently dies. ~100s of "Shell active" green status, then "Session ended" yellow with no error message. The fix: add -o ConnectTimeout=10 to ssh args so the failure surfaces as a real error within 10s instead of getting eaten by Cloudflare's idle-WS timeout.

Why this looked silent

  1. WebSocket upgrade succeeds → canvas shows "Connected to workspace shell" + green "Shell active"
  2. Server kicks off ssh-keygen → send-ssh-public-key → open-tunnel → ssh
  3. ssh hangs at TCP/SSH handshake to the workspace EC2's port 22 (sshd unresponsive)
  4. Pre-fix: ssh waits indefinitely for the banner. Workspace-server's local PTY (pty.Start slave) starts in cooked + echo mode, so the user's keystrokes get echoed back by the local PTY, masquerading as a working shell
  5. After ~100s, Cloudflare closes the WS for idle. Canvas → "Session ended" with no error frame
  6. Server-side log has the truth, but the user has no way to see it

Repro

$ python3 /tmp/term_probe.py  # holds a fresh terminal WS open for 60s
[t=0.0s] connected (101)
[t=8.0s] sending 'X' to test local PTY echo
[t=8.4s] BIN 1b: b'X'
# (no further frames for 52s — ssh still hung)

The single byte echoed back at t=8.4s is the workspace-server's local PTY echoing the keystroke before ssh ever connects. Confirms ssh is hung at handshake; bash on the workspace never sees it. Caught 2026-04-30 on hongmingwang hermes workspace 32993ee7-840e-4c02-8ca8-cb9d75d112a5 after the heartbeat-fix platform redeploy.

Fix

 var sshCommandCmd = func(o eicSSHOptions) *exec.Cmd {
   return exec.Command(
     "ssh",
     "-i", o.PrivateKeyPath,
     "-o", "StrictHostKeyChecking=no",
     "-o", "UserKnownHostsFile=/dev/null",
+    "-o", "ConnectTimeout=10",
     "-o", "ServerAliveInterval=30",
     "-o", "ServerAliveCountMax=3",
     ...

ssh now exits within 10s with a real error (ssh: connect to host 127.0.0.1 port X: Connection refused or similar). The PTY → WebSocket goroutine reads that error and forwards it to the canvas xterm before sending CloseMessage. User sees the actual reason within seconds.

This does NOT diagnose why sshd isn't responding — that's a separate investigation (likely SG/EIC misconfig or a workspace EC2 boot-flap). But it does mean that whatever the underlying cause, the user gets actionable feedback fast instead of looking at a silently dead terminal.

Test plan

  • New TestSSHCommandCmd_ConnectTimeoutPresent — behavior-based gate that asserts -o ConnectTimeout=N is in the ssh argv. Pins presence, not the literal value, so operators can tune without breaking the gate.
  • Negative-control verified: removing the line from terminal.go flips the test red with the expected message; restoring flips it green
  • Existing terminal_test.go suite (KI-005, GH#756) still passes — no behavioral regression in the auth path
  • Live verification on hongmingwang.moleculesai.app once auto-promote staging→main rolls

🤖 Generated with Claude Code

github-actions Bot and others added 22 commits April 30, 2026 04:29
When the workspace EC2's sshd is unresponsive (mid-restart, SG drop,
AMI without ec2-instance-connect), the canvas's xterm shows the user's
typed bytes echoed back by the workspace-server's *local* PTY (cooked +
echo mode before ssh sets it raw post-handshake) and then closes
silently when Cloudflare's idle WebSocket timer fires (~100s) — with no
"Connection refused" or "Permission denied" output ever reaching the
user. This is what hongmingwang's hermes terminal looked like 2026-04-30
right after the heartbeat-fix redeploy: status="online" but the shell
appeared dead.

Caught reproducibly by holding a fresh /workspaces/<id>/terminal
WebSocket open for 60s — server sent zero frames except the local-PTY
echo of one keystroke typed at t=8s. ssh was hung at handshake; bash
never saw the byte.

Fix: add `-o ConnectTimeout=10` to ssh args. Now the failure surfaces
as a real ssh error message in the terminal within 10s, instead of
masquerading as a silently dead shell over the next ~100s. Doesn't
diagnose *why* sshd isn't responding (separate investigation), but
it does mean the user gets actionable feedback within seconds.

Behavior-based regression test asserts `-o ConnectTimeout=N` is in the
ssh argv — pins presence, not the literal value, so operators can tune
without breaking the gate. Verified to FAIL on pre-fix code (matched
the literal arg pair) and PASS on fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pre-existing TestSSHCommandCmd_BuildsArgv asserts the literal argv
slice. Adding `-o ConnectTimeout=10` shifted the slice — this commit
tracks the snapshot to match. The new behavior-based
TestSSHCommandCmd_ConnectTimeoutPresent (added in the prior commit)
keeps the invariant pinned without depending on argv ordering, so
future tweaks land in only one place even if more options are added.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented May 1, 2026

Copy link
Copy Markdown
Contributor

🔒 Auto-merge disabled — new commit (cda93e3) pushed after auto-merge was enabled. The merge queue locks SHAs at entry, so subsequent pushes can race. Verify the new commit and re-enable with gh pr merge --auto.

@HongmingWang-Rabbit
HongmingWang-Rabbit added this pull request to the merge queue May 1, 2026
Merged via the queue into staging with commit 4e39609 May 1, 2026
21 checks passed
@HongmingWang-Rabbit
HongmingWang-Rabbit deleted the fix/terminal-ssh-connect-timeout branch May 1, 2026 03:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant