fix(terminal): cap ssh handshake at 10s so hung sshd surfaces fast - #2434
Merged
Conversation
staging → main: auto-promote 9d159f0
staging → main: auto-promote 21ed74c
staging → main: auto-promote 0b83faa
staging → main: auto-promote b5bde03
staging → main: auto-promote c1f993c
staging → main: auto-promote 92a29bb
staging → main: auto-promote 7cb8b47
staging → main: auto-promote cdef893
staging → main: auto-promote 904cf31
staging → main: auto-promote 36fd658
staging → main: auto-promote dea306d
staging → main: auto-promote 6159429
staging → main: auto-promote d2046c3
staging → main: auto-promote d00c8be
staging → main: auto-promote cc58e87
staging → main: auto-promote 665582b
staging → main: auto-promote f035482
staging → main: auto-promote c733454
staging → main: auto-promote 2a56697
staging → main: auto-promote 03d5f80
staging → main: auto-promote 0c51df9
When the workspace EC2's sshd is unresponsive (mid-restart, SG drop, AMI without ec2-instance-connect), the canvas's xterm shows the user's typed bytes echoed back by the workspace-server's *local* PTY (cooked + echo mode before ssh sets it raw post-handshake) and then closes silently when Cloudflare's idle WebSocket timer fires (~100s) — with no "Connection refused" or "Permission denied" output ever reaching the user. This is what hongmingwang's hermes terminal looked like 2026-04-30 right after the heartbeat-fix redeploy: status="online" but the shell appeared dead. Caught reproducibly by holding a fresh /workspaces/<id>/terminal WebSocket open for 60s — server sent zero frames except the local-PTY echo of one keystroke typed at t=8s. ssh was hung at handshake; bash never saw the byte. Fix: add `-o ConnectTimeout=10` to ssh args. Now the failure surfaces as a real ssh error message in the terminal within 10s, instead of masquerading as a silently dead shell over the next ~100s. Doesn't diagnose *why* sshd isn't responding (separate investigation), but it does mean the user gets actionable feedback within seconds. Behavior-based regression test asserts `-o ConnectTimeout=N` is in the ssh argv — pins presence, not the literal value, so operators can tune without breaking the gate. Verified to FAIL on pre-fix code (matched the literal arg pair) and PASS on fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit
requested a review
from hongmingwang-moleculeai
as a code owner
May 1, 2026 03:17
HongmingWang-Rabbit
enabled auto-merge
May 1, 2026 03:17
The pre-existing TestSSHCommandCmd_BuildsArgv asserts the literal argv slice. Adding `-o ConnectTimeout=10` shifted the slice — this commit tracks the snapshot to match. The new behavior-based TestSSHCommandCmd_ConnectTimeoutPresent (added in the prior commit) keeps the invariant pinned without depending on argv ordering, so future tweaks land in only one place even if more options are added. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
🔒 Auto-merge disabled — new commit ( |
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a workspace EC2's sshd is unresponsive (mid-restart, SG drop, AMI without ec2-instance-connect installed), the canvas Terminal tab silently dies. ~100s of "Shell active" green status, then "Session ended" yellow with no error message. The fix: add
-o ConnectTimeout=10to ssh args so the failure surfaces as a real error within 10s instead of getting eaten by Cloudflare's idle-WS timeout.Why this looked silent
pty.Startslave) starts in cooked + echo mode, so the user's keystrokes get echoed back by the local PTY, masquerading as a working shellRepro
The single byte echoed back at t=8.4s is the workspace-server's local PTY echoing the keystroke before ssh ever connects. Confirms ssh is hung at handshake; bash on the workspace never sees it. Caught 2026-04-30 on hongmingwang hermes workspace
32993ee7-840e-4c02-8ca8-cb9d75d112a5after the heartbeat-fix platform redeploy.Fix
var sshCommandCmd = func(o eicSSHOptions) *exec.Cmd { return exec.Command( "ssh", "-i", o.PrivateKeyPath, "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null", + "-o", "ConnectTimeout=10", "-o", "ServerAliveInterval=30", "-o", "ServerAliveCountMax=3", ...ssh now exits within 10s with a real error (
ssh: connect to host 127.0.0.1 port X: Connection refusedor similar). The PTY → WebSocket goroutine reads that error and forwards it to the canvas xterm before sending CloseMessage. User sees the actual reason within seconds.This does NOT diagnose why sshd isn't responding — that's a separate investigation (likely SG/EIC misconfig or a workspace EC2 boot-flap). But it does mean that whatever the underlying cause, the user gets actionable feedback fast instead of looking at a silently dead terminal.
Test plan
TestSSHCommandCmd_ConnectTimeoutPresent— behavior-based gate that asserts-o ConnectTimeout=Nis in the ssh argv. Pins presence, not the literal value, so operators can tune without breaking the gate.terminal.goflips the test red with the expected message; restoring flips it green🤖 Generated with Claude Code