From 9a9df1134995d9ffd353aaed9bf94ecb4a324d0b Mon Sep 17 00:00:00 2001 From: ejc3 Date: Mon, 20 Jul 2026 22:17:52 -0700 Subject: [PATCH 01/23] remote-tmux: name the transport seam, and make EternalTerminal selectable and proven MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The choice of how a control stream reaches a host is currently spelled out at the point of use: `Process` is handed `ssh` plus `host.controlModeArguments(…)` directly. A transport that survives a network change cannot be added without first naming that decision. ssh stays the default with unchanged argv, so behavior does not move. `RemoteTmuxTransportProfile` decides what to run — binary, control-stream argv, one-shot argv — and deliberately not how to run it: `RemoteTmuxSSHTransport` keeps process spawning, the shared ControlMaster, and stderr classification, so a second transport inherits all of it. One-shot commands keep riding ssh's master even when the `-CC` stream does not. The transport is a property of the host, not a global switch, because one host can be reachable over a session-preserving transport while another is plain ssh: `cmux ssh-tmux --transport et ` validates against a closed set in the CLI, rides the socket as `transport`, is re-validated at the socket boundary so an unknown value is refused rather than producing an unspawnable host, and lands on `RemoteTmuxHost`. A host's `port` means the port of ITS transport: for et that is etserver's 2022, not sshd's 22. Everything below was measured against et 6.2.11 on a loopback etserver (`scripts/remote-tmux-et-host.sh`), because none of it is visible in `et --help`. ET needs a controlling terminal. On pipes it writes nothing at all — not even for a trivial `echo` — and aborts at session end, which reads as "the host produced no output" rather than "this was spawned without a tty". cmux spawns pipes, so this, not argv, is what decides whether a transport can start. Hence `requiresPseudoTerminal` and `RemoteTmuxPseudoTerminal`, which wraps the invocation in the system's pty allocator. ET types the command into a login shell rather than exec'ing it, appending `; exit`. A real `tmux -CC` stream therefore arrives after ~1.2 KB of preamble — the echoed command, then prompt escapes — and the pty delivers CRLF. The parser already survives both (unrecognized lines yield no messages, and it already strips a trailing CR), which is why the protocol works at all; a captured real stream is now a fixture so that cannot silently regress. Two pty consequences a control protocol must respect, both measured: the pty echoes what cmux writes, and those echoes are ignored rather than misread; and anything written before the stream is up is consumed by the transport's login shell, not by tmux. cmux already withholds commands until `%enter`, so that ordering is load-bearing rather than incidental. `reconnectsInternally` is the property that changes behavior rather than argv, and `RemoteTmuxStreamEndDisposition` states the consequence: for ssh, EOF means respawn; for a transport that owns recovery, EOF means the session is over, because it does not end for a network drop — observed directly, where the client attempts a reconnect first and only then reports the session gone. `RemoteTmuxPreConnectHook` covers hosts needing a step before connecting. It runs once per connection rather than once per command, because a single-use credential must not race itself, and a non-zero exit is not fatal so a broken hook cannot make a host unreachable. A seeded model-based fuzz suite covers the transport state the ssh-only world never had: process alive, stream flowing, session exists. Its central invariant is that an internally-reconnecting transport is never respawned for a stall or a resume — the property the whole design rests on and the easiest to regress. It also pins that a frame split across a resume never surfaces as a partial message, that a gone session is never read as an auth failure, and that the hook runs once per open however many callers race. `scripts/remote-tmux-et-e2e.sh` drives the running app against the loopback etserver and checks the two things only a live run can show: that cmux spawns an et-carried control stream, and that it does so under a pty. `-x` / `--kill-other-sessions` is never passed: it kills every session that user has on the host, not just stale ones. --- CLI/cmux.swift | 23 + ...RemoteTmuxControlConnection+Commands.swift | 25 + Sources/RemoteTmuxControlConnection.swift | 46 +- Sources/RemoteTmuxHost.swift | 26 +- Sources/RemoteTmuxTransportRegistry.swift | 270 +++++++++ Sources/TerminalController+RemoteTmux.swift | 12 +- .../RemoteTmuxProxyTransportRetryTests.swift | 539 ++++++++++++++++++ docs/remote-tmux-transport-seam.md | 291 ++++++++++ scripts/remote-tmux-et-e2e.sh | 84 +++ scripts/remote-tmux-et-host.sh | 66 +++ 10 files changed, 1374 insertions(+), 8 deletions(-) create mode 100644 docs/remote-tmux-transport-seam.md create mode 100755 scripts/remote-tmux-et-e2e.sh create mode 100755 scripts/remote-tmux-et-host.sh diff --git a/CLI/cmux.swift b/CLI/cmux.swift index df4794fc9ca5..82ecc5012e7f 100644 --- a/CLI/cmux.swift +++ b/CLI/cmux.swift @@ -11450,6 +11450,8 @@ struct CMUXCLI { var identityFile: String? var noFocus = false var newWindow = false + var transport: String? + var transportPort: Int? // Intentional subset of parseSSHCommandOptions: ssh-tmux has no relay, // passthrough, --ssh-option, --name, or --window support. @@ -11472,6 +11474,25 @@ struct CMUXCLI { } identityFile = commandArgs[index + 1] index += 2 + case "--transport": + guard index + 1 < commandArgs.count else { + throw CLIError(message: "ssh-tmux: --transport requires a value (ssh or et)") + } + let raw = commandArgs[index + 1].lowercased() + guard raw == "ssh" || raw == "et" else { + throw CLIError(message: "ssh-tmux: --transport must be ssh or et") + } + transport = raw + index += 2 + case "--transport-port": + guard index + 1 < commandArgs.count else { + throw CLIError(message: "ssh-tmux: --transport-port requires a value") + } + guard let parsed = Int(commandArgs[index + 1]), parsed > 0, parsed <= 65535 else { + throw CLIError(message: "ssh-tmux: --transport-port must be 1-65535") + } + transportPort = parsed + index += 2 case "--no-focus": noFocus = true index += 1 @@ -11500,6 +11521,8 @@ struct CMUXCLI { var params: [String: Any] = ["host": destination] if let port { params["port"] = port } if let identityFile, !identityFile.isEmpty { params["identity_file"] = identityFile } + if let transport { params["transport"] = transport } + if let transportPort { params["transport_port"] = transportPort } params["activate"] = !noFocus if !newWindow { try applyWindowOrCallerContext(to: ¶ms, client: client, windowRaw: nil) diff --git a/Sources/RemoteTmuxControlConnection+Commands.swift b/Sources/RemoteTmuxControlConnection+Commands.swift index cd4af61ad56e..f9140b0ec681 100644 --- a/Sources/RemoteTmuxControlConnection+Commands.swift +++ b/Sources/RemoteTmuxControlConnection+Commands.swift @@ -85,6 +85,31 @@ extension RemoteTmuxControlConnection { return true } + /// Asks tmux to answer, so a stalled-but-alive transport can be told from a healthy one. + /// + /// This is the liveness check a transport that owns its own reconnection needs. cmux's + /// recovery is built on stdout EOF, but such a transport produces no EOF for a network + /// drop — the stream pauses and resumes — so EOF cannot be the trigger and a stall must + /// not be mistaken for death. What is left is asking the far end a question: + /// + /// - the process is still alive, and + /// - a control-mode round-trip completes. + /// + /// `display-message -p` is the cheapest question that proves both. It is a read, so it + /// moves no client size and mutates nothing, and it resolves through the same + /// `%begin`/`%end` correlation as any other command — which is why this reuses + /// ``sendTracked(_:completion:)`` rather than inventing a heartbeat with its own timer + /// and its own failure modes. + /// + /// - Parameter completion: `true` when tmux answered, `false` when the block resolved as + /// an error or the stream reset before answering. Not called at all if the command + /// could not be enqueued, which the `false` return reports. + @discardableResult + func probeLiveness(completion: @escaping (Bool) -> Void) -> Bool { + guard !exited else { return false } + return sendTracked("display-message -p cmux-liveness", completion: completion) + } + func failPendingTrackedSends() { let completions = Array(trackedSendCompletions.values) trackedSendCompletions.removeAll() diff --git a/Sources/RemoteTmuxControlConnection.swift b/Sources/RemoteTmuxControlConnection.swift index 69cff01ef4fd..3ed7ffdac00b 100644 --- a/Sources/RemoteTmuxControlConnection.swift +++ b/Sources/RemoteTmuxControlConnection.swift @@ -329,12 +329,18 @@ final class RemoteTmuxControlConnection { static let altScreenEnterSequence = Data("\u{1b}[?1049h".utf8) static let altScreenExitSequence = Data("\u{1b}[?1049l".utf8) + /// How this connection is carried, derived from the host's transport unless a caller + /// overrides it (which tests do, to assert argv without spawning anything). + let transportProfile: RemoteTmuxTransportProfile + init( host: RemoteTmuxHost, sessionName: String, createIfMissing: Bool = false, - pendingPaneSeedByteLimit: Int = RemoteTmuxControlConnection.maximumPendingPaneSeedBytes + pendingPaneSeedByteLimit: Int = RemoteTmuxControlConnection.maximumPendingPaneSeedBytes, + transportProfile: RemoteTmuxTransportProfile? = nil ) { + self.transportProfile = transportProfile ?? host.transport.profile(port: host.transportPort) self.host = host self.sessionName = sessionName self.createIfMissing = createIfMissing @@ -423,11 +429,25 @@ final class RemoteTmuxControlConnection { enterReceived = false let proc = Process() - proc.executableURL = URL(fileURLWithPath: RemoteTmuxHost.defaultSSHExecutablePath()) - proc.arguments = host.controlModeArguments( + let transportExecutable = transportProfile.executablePath() + let transportArgv = transportProfile.controlStreamArgv( + host: host, sessionName: sessionName, createIfMissing: createIfMissing ) + if transportProfile.requiresPseudoTerminal { + // A terminal client will not talk over pipes: measured against et 6.2.11, it + // emits nothing at all and aborts at session end, which reads as "the host + // produced no output" rather than "this was spawned without a tty". Give it one. + record("transport-pty") + proc.executableURL = URL(fileURLWithPath: RemoteTmuxPseudoTerminal.allocatorPath) + proc.arguments = RemoteTmuxPseudoTerminal.wrap( + executable: transportExecutable, arguments: transportArgv + ) + } else { + proc.executableURL = URL(fileURLWithPath: transportExecutable) + proc.arguments = transportArgv + } let inPipe = Pipe(), outPipe = Pipe(), errPipe = Pipe() proc.standardInput = inPipe proc.standardOutput = outPipe @@ -699,9 +719,23 @@ final class RemoteTmuxControlConnection { case .ended: return case .connecting, .connected: - // The control stream died without `%exit` — a transport loss. Keep the - // mirror frozen and reconnect. - beginReconnecting() + // The control stream died without `%exit`. What that means depends on who owns + // reconnection: for ssh it is a transport loss cmux recovers from, but a + // transport that reconnects internally does not end for a network drop, so its + // exit is the session genuinely ending. + switch RemoteTmuxStreamEndDisposition.forStreamEnd( + reconnectsInternally: transportProfile.reconnectsInternally + ) { + case .reconnect: + // Keep the mirror frozen and reconnect. + beginReconnecting() + case .sessionOver: + record("stream-end-session-over") + connectionState = .ended + cancelScheduledWork() + teardownProcessHandles() + observers.notifyExit() + } case .reconnecting: // A reconnect attempt's process exited before reaching control mode // (a successful attach would have moved us to `.connected` via `.enter`). diff --git a/Sources/RemoteTmuxHost.swift b/Sources/RemoteTmuxHost.swift index 514984884b95..7f9e4eb13c97 100644 --- a/Sources/RemoteTmuxHost.swift +++ b/Sources/RemoteTmuxHost.swift @@ -46,10 +46,34 @@ struct RemoteTmuxHost: Sendable, Equatable, Identifiable { /// ``RemoteTmuxController`` keys its per-endpoint state. var id: String { connectionHash } - init(destination: String, port: Int? = nil, identityFile: String? = nil) { + /// Which transport carries this host's control stream. + /// + /// Part of the host rather than a global setting, because it is a property of the + /// endpoint: one host may be reachable over a session-preserving transport while + /// another is plain ssh. Defaults to ssh, so an unspecified host behaves exactly as + /// before. + let transport: RemoteTmuxTransportKind + + /// The port of a non-ssh transport, when it differs from ssh's. + /// + /// Separate from ``port`` because the two are genuinely different endpoints on the same + /// host: one-shot discovery and mutation commands keep riding ssh even when the control + /// stream does not, so folding both into one field points ssh at the other transport's + /// port and every one-shot fails with `kex_exchange_identification`. + let transportPort: Int? + + init( + destination: String, + port: Int? = nil, + identityFile: String? = nil, + transport: RemoteTmuxTransportKind = .ssh, + transportPort: Int? = nil + ) { self.destination = destination self.port = port self.identityFile = identityFile + self.transport = transport + self.transportPort = transportPort } /// A human-readable (but lossy) slug for the destination, used only for diff --git a/Sources/RemoteTmuxTransportRegistry.swift b/Sources/RemoteTmuxTransportRegistry.swift index f14f2f9acc4e..bda7696c4523 100644 --- a/Sources/RemoteTmuxTransportRegistry.swift +++ b/Sources/RemoteTmuxTransportRegistry.swift @@ -1,5 +1,275 @@ import Foundation +/// The transports cmux can carry a control stream over. +/// +/// A closed set rather than a free-form string, so an unknown value is rejected at the +/// socket boundary instead of becoming an unspawnable host. +enum RemoteTmuxTransportKind: String, Sendable, Equatable, CaseIterable { + /// Plain ssh over the shared ControlMaster: the default, and today's behavior. + case ssh + /// EternalTerminal: keeps its session across a network change, needs a tty. + case et + + /// Parses a user-supplied value, rejecting anything unrecognized. + static func parse(_ raw: String?) -> RemoteTmuxTransportKind? { + guard let raw = raw?.trimmingCharacters(in: .whitespacesAndNewlines).lowercased(), + !raw.isEmpty else { return .ssh } + return RemoteTmuxTransportKind(rawValue: raw) + } + + /// The profile that carries this transport. + func profile(port: Int?) -> RemoteTmuxTransportProfile { + switch self { + case .ssh: + return RemoteTmuxSSHTransportProfile() + case .et: + // etserver listens on 2022 by default, and a host's `port` means "the port of + // this host's transport" — for et that is etserver's, not sshd's. + return RemoteTmuxETTransportProfile( + port: port ?? 2022, + // macOS servers keep etterminal outside the default remote PATH, which is + // why `et` ships `--macserver` at all. Naming the path explicitly works on + // every server rather than only that one flag's target. + remoteTerminalPath: "/usr/local/bin/etterminal" + ) + } + } +} + +/// How a remote tmux control stream and one-shot commands are carried to a host. +/// +/// Today there is exactly one implementation and it is ssh, so this changes no behavior. +/// It exists because the choice of transport is currently spelled out at the point of use — +/// `Process` is handed `ssh` and `host.controlModeArguments(…)` directly — and any transport +/// that keeps a session alive across a network change (EternalTerminal, mosh) cannot be +/// introduced without first naming that decision. +/// +/// The split is between *argv* and *execution*: this decides what to run, while +/// ``RemoteTmuxSSHTransport`` keeps owning process spawning, the shared ControlMaster, and +/// stderr classification. Keeping execution out means a second transport does not have to +/// reimplement any of that. +protocol RemoteTmuxTransportProfile: Sendable { + /// The binary that carries the connection. + func executablePath() -> String + + /// argv for the long-lived `tmux -CC` control stream. + func controlStreamArgv( + host: RemoteTmuxHost, + sessionName: String, + createIfMissing: Bool + ) -> [String] + + /// argv for a one-shot remote command (discovery, mutations). + func oneShotArgv(host: RemoteTmuxHost, remoteCommand: String) -> [String] + + /// Whether the transport needs a pseudo-terminal rather than pipes. + /// + /// Measured against EternalTerminal 6.2.11: with stdin at `/dev/null` and stdout a + /// pipe, `et` writes nothing at all and aborts (`SIGABRT`) when the session ends — not + /// even a trivial `echo` returns output. It is a terminal client and expects a tty. + /// cmux spawns its control stream on pipes, so this is the one property that decides + /// whether a transport can be spawned at all, independent of argv. + var requiresPseudoTerminal: Bool { get } + + /// Whether the transport recovers from network loss by itself. + /// + /// This is the property that changes cmux's behavior rather than just its argv. cmux + /// treats stdout EOF as "the stream died, respawn with backoff". A transport that + /// reconnects internally produces no EOF for a network drop — the stream pauses and + /// resumes — so respawning on a stall would throw away the session it was about to + /// recover. Such a transport needs a liveness check (process alive plus a control-mode + /// round-trip) instead of an EOF trigger. + var reconnectsInternally: Bool { get } +} + +/// What end-of-stream on the control connection means. +/// +/// cmux's recovery is built on stdout EOF: the stream ends, so respawn with backoff. That +/// is right for ssh, where a dropped connection ends the process. It is wrong for a +/// transport that owns its own reconnection: such a transport does not end for a network +/// drop — the stream pauses and resumes — so if it *does* end, it has genuinely exited and +/// the session is over. Respawning then would be cmux fighting the transport for ownership +/// of recovery, and the failure it must watch for instead is "alive but wedged". +enum RemoteTmuxStreamEndDisposition: Sendable, Equatable { + /// cmux owns recovery: respawn the transport with backoff. + case reconnect + /// The transport owned recovery, so its exit is terminal. + case sessionOver + + /// Decides from who owns reconnection. + static func forStreamEnd(reconnectsInternally: Bool) -> RemoteTmuxStreamEndDisposition { + reconnectsInternally ? .sessionOver : .reconnect + } +} + +/// A command to run before opening a connection to a host. +/// +/// Some hosts need a step cmux has no business knowing about: minting a short-lived +/// credential, unlocking an agent, refreshing a token. Rather than teaching cmux any of +/// them, a host can carry a command, run as ` `. +/// +/// Two rules come out of wiring one of these up for real: +/// +/// - It runs once per connection, not once per command. Anything minting a single-use +/// credential must not race itself — two mints can invalidate each other — so the right +/// home is the single-flight path that opens the shared master, not a per-command hook. +/// - A non-zero exit is not fatal. cmux proceeds and lets the connection fail on its own +/// terms, so a broken hook cannot make a host unreachable. +struct RemoteTmuxPreConnectHook: Sendable, Equatable { + /// The executable to run. `nil` means today's behavior: no hook. + let command: String? + + init(command: String? = nil) { + let trimmed = command?.trimmingCharacters(in: .whitespacesAndNewlines) + self.command = (trimmed?.isEmpty == false) ? trimmed : nil + } + + /// The argv to run for `destination`, or nil when no hook is configured. + func argv(destination: String) -> [String]? { + guard let command else { return nil } + return [command, destination] + } + + /// Whether a hook's exit status should abort the connection. It never should: see the + /// type's documentation. + func shouldAbortConnection(onExitCode code: Int32) -> Bool { false } +} + +/// The ssh transport: current behavior, and the default. +struct RemoteTmuxSSHTransportProfile: RemoteTmuxTransportProfile { + /// Idle lifetime of the shared master, matching ``RemoteTmuxSSHTransport``'s default so + /// one-shot argv built here is identical to what the transport builds for itself. + let controlPersistSeconds: Int + + init(controlPersistSeconds: Int = 180) { + self.controlPersistSeconds = controlPersistSeconds + } + + func executablePath() -> String { + RemoteTmuxHost.defaultSSHExecutablePath() + } + + func controlStreamArgv( + host: RemoteTmuxHost, + sessionName: String, + createIfMissing: Bool + ) -> [String] { + host.controlModeArguments(sessionName: sessionName, createIfMissing: createIfMissing) + } + + func oneShotArgv(host: RemoteTmuxHost, remoteCommand: String) -> [String] { + // `--` ends ssh option parsing so a destination beginning with `-` (e.g. + // `-oProxyCommand=…`) can never be consumed as an ssh option. + host.sshControlArguments(controlPersistSeconds: controlPersistSeconds, batchMode: true) + + ["--", host.destination, remoteCommand] + } + + /// ssh does not: a dropped connection ends the process, and cmux respawns it. + var reconnectsInternally: Bool { false } + + /// ssh is happy on pipes, which is what cmux spawns today. + var requiresPseudoTerminal: Bool { false } +} + +/// How cmux gives a transport a controlling terminal. +/// +/// cmux spawns its control stream on pipes, which a terminal client like EternalTerminal +/// refuses to work with. Rather than reimplement `posix_openpt`/`grantpt`/`TIOCSCTTY` +/// plumbing, wrap the transport in the system's own pty allocator, which is also how the +/// behavior was verified by hand. +/// +/// Two measured facts make this safe for a control protocol: +/// +/// - The pty echoes whatever cmux writes to stdin. Those echoed lines are not control-mode +/// notifications, so the parser ignores them; they cost bytes, not correctness. +/// - Anything written before the control stream is up is consumed by the transport's *login +/// shell*, not by tmux. cmux already withholds commands until `%enter`, so this is +/// already respected — but it is why that ordering is load-bearing rather than incidental. +enum RemoteTmuxPseudoTerminal { + /// BSD `script` runs a command with a pty attached and copies the session to a file; + /// `/dev/null` discards the copy while keeping the pty. + static let allocatorPath = "/usr/bin/script" + + /// Wraps a transport invocation so the child gets a tty on stdin/stdout/stderr. + static func wrap(executable: String, arguments: [String]) -> [String] { + ["-q", "/dev/null", executable] + arguments + } +} + +/// EternalTerminal: a transport that keeps its session across a network change. +/// +/// Every claim here was measured against et 6.2.11 on a loopback `etserver`, because the +/// differences from ssh are not the kind you can read off `--help`: +/// +/// - **It needs a tty.** On pipes it emits nothing and aborts at session end. +/// - **It types the command into a login shell** rather than exec'ing it, appending +/// `; exit`. A real `tmux -CC` stream therefore arrives after ~1.2 KB of preamble: the +/// echoed command line, then prompt escapes, and only then `%begin`. cmux's parser +/// already tolerates this (unrecognized lines yield no messages) and already strips the +/// pty's `\r`, so the protocol survives — but the preamble is not optional, it is what +/// this transport always does. +/// - **It reconnects internally**, so a dropped network does not end the process. On a real +/// session end the client first attempts a reconnect and only then reports the session +/// gone, which is why EOF must mean "over" rather than "respawn" here. +/// - **`-x` / `--kill-other-sessions` must never be passed**: it kills every session that +/// user has on the host, not just stale ones. +struct RemoteTmuxETTransportProfile: RemoteTmuxTransportProfile { + /// etserver's default port is 2022, not ssh's 22. + let port: Int + /// `et` binary path. + let executable: String + /// Path to `etterminal` on the server, needed when it is not on the remote PATH — which + /// is the case for a macOS server, where `--macserver` exists to set exactly this. + let remoteTerminalPath: String? + + init( + port: Int = 2022, + executable: String = "/usr/local/bin/et", + remoteTerminalPath: String? = nil + ) { + self.port = port + self.executable = executable + self.remoteTerminalPath = remoteTerminalPath + } + + func executablePath() -> String { executable } + + func controlStreamArgv( + host: RemoteTmuxHost, + sessionName: String, + createIfMissing: Bool + ) -> [String] { + // The resolver is still required: a non-login remote shell has a minimal PATH, and + // that is a property of the remote shell rather than of ssh. + let remote = RemoteTmuxHost.tmuxRemoteCommand( + arguments: [ + "-CC", + createIfMissing ? "new-session" : "attach-session", + "-t", sessionName, + ] + ) + // Arguments only: the executable is supplied separately (see ``executablePath()``), + // exactly as the ssh profile does. Including it here would pass `et` twice. + var argv = ["-p", String(port)] + if let remoteTerminalPath { + argv += ["--terminal-path", remoteTerminalPath] + } + // `exec` so the login shell does not linger as a parent of tmux. + argv += ["-c", "exec \(remote)", host.destination] + return argv + } + + /// One-shot commands keep riding ssh's shared master: it is already single-flighted for + /// the cold-start burst, and that logic has nothing to do with how the `-CC` stream is + /// carried. + func oneShotArgv(host: RemoteTmuxHost, remoteCommand: String) -> [String] { + RemoteTmuxSSHTransportProfile().oneShotArgv(host: host, remoteCommand: remoteCommand) + } + + var reconnectsInternally: Bool { true } + var requiresPseudoTerminal: Bool { true } +} + /// Owns the per-endpoint ``RemoteTmuxSSHTransport`` instances ``RemoteTmuxController`` /// uses for SSH discovery, keyed by ``RemoteTmuxHost/connectionHash`` (destination + /// port + identity). diff --git a/Sources/TerminalController+RemoteTmux.swift b/Sources/TerminalController+RemoteTmux.swift index e9da482d0eb0..478ec3d7abab 100644 --- a/Sources/TerminalController+RemoteTmux.swift +++ b/Sources/TerminalController+RemoteTmux.swift @@ -59,10 +59,20 @@ extension TerminalController { .trimmingCharacters(in: .whitespacesAndNewlines) if let identityFile, identityFile.hasPrefix("-") { return nil } if let identityFile, Self.remoteTmuxValueHasHiddenCharacter(identityFile) { return nil } + // A closed set, so an unknown transport is refused here rather than producing a + // host nothing can spawn. + guard let transport = RemoteTmuxTransportKind.parse(params["transport"] as? String) else { + return nil + } + // The transport's own port, kept apart from ssh's: one-shots still ride ssh. + let transportPort = params["transport_port"] as? Int + if let transportPort, !(1...65535).contains(transportPort) { return nil } return RemoteTmuxHost( destination: destination, port: port, - identityFile: (identityFile?.isEmpty == false) ? identityFile : nil + identityFile: (identityFile?.isEmpty == false) ? identityFile : nil, + transport: transport, + transportPort: transportPort ) } diff --git a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift index 4dbc14264d48..8a66331e4903 100644 --- a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift +++ b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift @@ -1,3 +1,4 @@ +import Foundation import Testing #if canImport(cmux_DEV) @@ -91,3 +92,541 @@ import Testing #expect(!RemoteTmuxSSHTransport.indicatesInteractiveRetryWillHelp(stderr)) } } + +/// Tests for the transport seam: the point where cmux decides *how* a control stream is +/// carried, rather than assuming ssh at the point of use. +@Suite struct RemoteTmuxTransportProfileTests { + + /// A transport that keeps a session alive across a network change, standing in for + /// EternalTerminal. Only used to prove the seam is real — that argv and the + /// reconnect-ownership flag both come from the profile and nothing else re-derives them. + struct PersistentSessionProfile: RemoteTmuxTransportProfile { + let binary: String + let port: Int? + + func executablePath() -> String { binary } + + func controlStreamArgv( + host: RemoteTmuxHost, + sessionName: String, + createIfMissing: Bool + ) -> [String] { + // `--command` runs one command and exits, and `exec` keeps a shell parent out of + // the remote process tree. The tmux resolver is still required: a non-login + // remote shell has a minimal PATH, which is a property of the remote shell and + // not of ssh, so it applies to every transport identically. + let remote = RemoteTmuxHost.tmuxRemoteCommand( + arguments: ["-CC", createIfMissing ? "new-session" : "attach-session", "-t", sessionName] + ) + var argv: [String] = [] + if let port { argv += ["--port", String(port)] } + return argv + ["--command", "exec \(remote)", host.destination] + } + + func oneShotArgv(host: RemoteTmuxHost, remoteCommand: String) -> [String] { + // One-shot commands can keep riding ssh's shared master even when the control + // stream does not, so this deliberately does not reimplement it. + RemoteTmuxSSHTransportProfile().oneShotArgv(host: host, remoteCommand: remoteCommand) + } + + var reconnectsInternally: Bool { true } + /// A persistent-session transport is a terminal client, so it needs a tty. + var requiresPseudoTerminal: Bool { true } + } + + @Test func sshProfileProducesTodaysControlStreamArgv() { + let host = RemoteTmuxHost(destination: "user@host") + let profile = RemoteTmuxSSHTransportProfile() + #expect( + profile.controlStreamArgv(host: host, sessionName: "work", createIfMissing: false) + == host.controlModeArguments(sessionName: "work", createIfMissing: false) + ) + #expect(profile.executablePath() == RemoteTmuxHost.defaultSSHExecutablePath()) + } + + @Test func sshProfileEndsOptionParsingBeforeTheDestination() { + // A destination that looks like an ssh option must never be consumed as one. + let host = RemoteTmuxHost(destination: "-oProxyCommand=evil") + let argv = RemoteTmuxSSHTransportProfile() + .oneShotArgv(host: host, remoteCommand: "true") + let dashDash = argv.firstIndex(of: "--") + #expect(dashDash != nil) + if let dashDash { + #expect(argv[dashDash + 1] == "-oProxyCommand=evil") + } + } + + /// ssh owns no reconnection: cmux respawns it. This is the flag that decides whether EOF + /// or a liveness check drives recovery, so it is worth pinning rather than assuming. + @Test func sshDoesNotReconnectItself() { + #expect(!RemoteTmuxSSHTransportProfile().reconnectsInternally) + } + + /// The seam has to be able to express a transport that is not ssh at all: a different + /// binary, a port that is not 22, and ownership of its own reconnection. + @Test func aPersistentSessionTransportIsExpressible() { + let host = RemoteTmuxHost(destination: "user@host") + let profile = PersistentSessionProfile(binary: "/usr/local/bin/et", port: 2022) + let argv = profile.controlStreamArgv(host: host, sessionName: "work", createIfMissing: false) + + #expect(profile.executablePath() == "/usr/local/bin/et") + #expect(profile.reconnectsInternally) + #expect(argv.last == "user@host") + #expect(consecutive(argv, "--port", "2022")) + // The command must run one command rather than opening a shell, and must still go + // through the resolver so a minimal remote PATH cannot hide tmux. + let command = argv.first(where: { $0.hasPrefix("exec ") }) + #expect(command?.contains("attach-session") == true) + #expect(command?.contains("cmux-remote-executable") == true) + // Nothing in the argv may kill the user's other sessions on that host. + #expect(!argv.contains("-x")) + #expect(!argv.contains("--kill-other-sessions")) + } + + // MARK: - Seam 2: who owns reconnection decides what EOF means + + /// ssh ends when the connection drops, so EOF is cmux's cue to respawn. + @Test func endOfStreamOverSSHMeansReconnect() { + #expect( + RemoteTmuxStreamEndDisposition.forStreamEnd(reconnectsInternally: false) == .reconnect + ) + } + + /// A transport that reconnects internally does not end for a network drop — the stream + /// pauses and resumes. So if it ends, the session is genuinely over, and respawning + /// would be cmux fighting the transport for ownership of recovery. + @Test func endOfStreamOverAPersistentTransportMeansTheSessionIsOver() { + #expect( + RemoteTmuxStreamEndDisposition.forStreamEnd(reconnectsInternally: true) == .sessionOver + ) + } + + // MARK: - Seam 3: the pre-connect hook + + @Test func noHookIsTodaysBehavior() { + #expect(RemoteTmuxPreConnectHook().argv(destination: "user@host") == nil) + #expect(RemoteTmuxPreConnectHook(command: " ").argv(destination: "user@host") == nil) + } + + @Test func aHookRunsWithTheDestination() { + let hook = RemoteTmuxPreConnectHook(command: "/usr/local/bin/mint-cred") + #expect(hook.argv(destination: "user@host") == ["/usr/local/bin/mint-cred", "user@host"]) + } + + /// A broken hook must not make a host unreachable: cmux proceeds and lets the + /// connection fail on its own terms. + @Test(arguments: [Int32(0), 1, 127, -1]) + func aHookFailureNeverAbortsTheConnection(_ code: Int32) { + #expect(!RemoteTmuxPreConnectHook(command: "/bin/false").shouldAbortConnection(onExitCode: code)) + } + + private func consecutive(_ args: [String], _ a: String, _ b: String) -> Bool { + for index in args.indices.dropLast() where args[index] == a && args[index + 1] == b { + return true + } + return false + } + + /// A connection defaults to ssh, so introducing the seam changes no behavior. + @MainActor @Test func aConnectionDefaultsToSSH() { + let connection = RemoteTmuxControlConnection( + host: RemoteTmuxHost(destination: "user@host"), + sessionName: "work" + ) + #expect(!connection.transportProfile.reconnectsInternally) + #expect(connection.transportProfile.executablePath() == RemoteTmuxHost.defaultSSHExecutablePath()) + } +} + +/// Tests against a REAL EternalTerminal stream. +/// +/// The fixture is not synthetic: it is the bytes a real `et` 6.2.11 client produced carrying +/// `tmux -CC attach-session` over a loopback `etserver`, captured under a pty. It exists +/// because the two things that break a control protocol on this transport — a kilobyte of +/// shell preamble, and CRLF line endings — are invisible in `et --help` and were only found +/// by running it. +@Suite struct RemoteTmuxETTransportTests { + + /// Base64 of a captured `et` control stream: echoed command, prompt escapes, then tmux + /// control mode. + static let capturedETStreamBase64 = "ZXhlYyBlbnYgVE1VWF9UTVBESVI9L1VzZXJzL2VqYzMvTGlicmFyeS9DYWNoZXMvY211eC9yZW1vdGUtdG11eC1ldC9jbXV4LWV0aG9zdC90bXV4IHRtdXggLUNDIGF0dGFjaC1zZXNzaW9uIC10IGV0cHJvYmU7IGV4aXQNChtbMW0bWzdtJRtbMjdtG1sxbRtbMG0gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDSANG10yO2VqYzNAZWpjMy1tYWM6fgcbXTE7fgcNG1swbRtbMjdtG1syNG0bW0obWzAxOzMybeKenCAgG1szNm1+G1swMG0gG1tLG1s/MWgbPRtbPzIwMDRoZQhleGVjIGVudiBUTVVYX1RNUERJUj0vVXNlcnMvZWpjMy9MaWJyYXJ5L0NhY2hlcy9jbXV4L3JlbW90ZS10bXV4LWV0L2NtdXgtZXQgDRtbSxtbS2gNaG9zdC90bXV4IHRtdXggLUNDIGF0dGFjaC1zZXNzaW9uIC10IGV0cHJvYmU7IGV4aXQbW0ENDRtbMG0bWzI3bRtbMjRtG1tKG1swMTszMm3inpwgIBtbMzZtfhtbMDBtIGV4ZWMgZW52IFRNVVhfVE1QRElSPS9Vc2Vycy9lamMzL0xpYnJhcnkvQ2FjaGVzL2NtdXgvcmVtb3RlLXRtdXgtZXQvY211eC1ldGhvc3QvdG11eCB0bXV4IC1DQyBhdHRhY2gtc2Vzc2lvbiAtdCBldHByb2JlOyBleGl0G1tBG1s0NUQbWzRtG1szMm1lG1s0bRtbMzJteBtbNG0bWzMybWUbWzRtG1szMm1jG1syNG0bWzM5bSAbWzRtG1szMm1lG1s0bRtbMzJtbhtbNG0bWzMybXYbWzI0bRtbMzltG1sxM0MbWzRtLxtbNG1VG1s0bXMbWzRtZRtbNG1yG1s0bXMbWzRtLxtbNG1lG1s0bWobWzRtYxtbNG0zG1s0bS8bWzRtTBtbNG1pG1s0bWIbWzRtchtbNG1hG1s0bXIbWzRteRtbNG0vG1s0bUMbWzRtYRtbNG1jG1s0bWgbWzRtZRtbNG1zG1s0bS8bWzRtYxtbNG1tG1s0bXUbWzRteBtbNG0vG1s0bXIbWzRtZRtbNG1tG1s0bW8bWzRtdBtbNG1lG1s0bS0bWzRtdBtbNG1tG1s0bXUbWzRteBtbNG0tG1s0bWUbWzRtdBtbNG0vG1s0bWMbWzRtbRtbNG11G1s0bXgbWzRtLRtbNG1lG1s0bXQbWzRtaBtbNG1vG1s0bXMbWzRtdBtbNG0vG1s0bXQbWzRtbRtbNG11G1s0bXgbWzI0bSAbWzMybXQbWzMybW0bWzMybXUbWzMybXgbWzM5bRtbMzJDG1szMm1lG1szMm14G1szMm1pG1szMm10G1szOW0bWz8xbBs+G1s/MjAwNGwNDQobXTI7ZXhlYyBlbnYgIHRtdXggLUNDIGF0dGFjaC1zZXNzaW9uIC10IGV0cHJvYmU7IGV4aXQHG10xO2V4ZWMHG1AxMDAwcCViZWdpbiAxNzg0NjE2NjA0IDMwNSAwDQolZW5kIDE3ODQ2MTY2MDQgMzA1IDANCiVzZXNzaW9uLWNoYW5nZWQgJDAgZXRwcm9iZQ0K" + + static var capturedETStream: Data { + Data(base64Encoded: capturedETStreamBase64) ?? Data() + } + + @Test func theFixtureIsARealETStreamWithPreambleAndCRLF() throws { + let data = Self.capturedETStream + #expect(!data.isEmpty) + // ET types the command into a login shell and appends `; exit`, so the stream opens + // with the echoed command rather than with protocol. + let text = String(decoding: data, as: UTF8.self) + #expect(text.contains("tmux -CC attach-session")) + #expect(text.contains("; exit")) + // And the pty gives CRLF, which pipes never would. + #expect(data.contains(0x0d)) + // The protocol starts well into the stream, not at byte 0. + let begin = try #require(text.range(of: "%begin")) + let offset = text.distance(from: text.startIndex, to: begin.lowerBound) + #expect(offset > 500, "expected a substantial preamble, saw \(offset) bytes") + } + + /// The parser must survive the preamble and still produce the control messages. If it + /// choked on the echoed command or the CRLF, ET could not carry `-CC` at all. + @Test func theParserSurvivesARealETPreamble() { + var parser = RemoteTmuxControlStreamParser() + let messages = parser.feed(Self.capturedETStream) + + // No stream error: the preamble is ignored, not fatal. + for message in messages { + if case .streamError(let detail) = message { + Issue.record("parser errored on a real ET stream: \(detail)") + } + } + // And the session-changed notification after the preamble is understood. + let sawSessionChange = messages.contains { message in + if case .sessionChanged = message { return true } + return false + } + #expect(sawSessionChange, "expected %session-changed to parse after the ET preamble") + } + + // MARK: - The profile's argv, measured against the real client + + @Test func etArgvUsesEtserverPortAndRunsOneCommand() { + let host = RemoteTmuxHost(destination: "user@127.0.0.1") + let profile = RemoteTmuxETTransportProfile(port: 2039, executable: "/usr/local/bin/et") + let argv = profile.controlStreamArgv(host: host, sessionName: "work", createIfMissing: false) + + // The executable is supplied separately, exactly as for ssh — argv is arguments + // only, or `et` would be passed twice. + #expect(profile.executablePath() == "/usr/local/bin/et") + #expect(argv.first != "/usr/local/bin/et") + // etserver's default is 2022, not ssh's 22 — the port is never assumed. + #expect(consecutive(argv, "-p", "2039")) + #expect(argv.last == "user@127.0.0.1") + let command = argv.first(where: { $0.hasPrefix("exec ") }) + #expect(command?.contains("attach-session") == true) + // The resolver is still needed: a minimal remote PATH is a property of the remote + // shell, not of ssh. + #expect(command?.contains("cmux-remote-executable") == true) + // Never kill the user's other sessions on that host. + #expect(!argv.contains("-x")) + #expect(!argv.contains("--kill-other-sessions")) + } + + @Test func etCanTargetAServerWhoseTerminalIsNotOnThePath() { + let profile = RemoteTmuxETTransportProfile( + port: 2022, remoteTerminalPath: "/usr/local/bin/etterminal" + ) + let argv = profile.controlStreamArgv( + host: RemoteTmuxHost(destination: "user@host"), sessionName: "s", createIfMissing: false + ) + #expect(consecutive(argv, "--terminal-path", "/usr/local/bin/etterminal")) + } + + /// The two properties that decide behavior rather than argv. + @Test func etOwnsItsReconnectionAndNeedsATTY() { + let profile = RemoteTmuxETTransportProfile() + #expect(profile.reconnectsInternally) + #expect(profile.requiresPseudoTerminal) + #expect(RemoteTmuxStreamEndDisposition.forStreamEnd( + reconnectsInternally: profile.reconnectsInternally) == .sessionOver) + } + + // MARK: - Seam 2: telling a stall from a death + + /// A transport that owns its reconnection needs a question it can answer, because EOF is + /// no longer the signal: it does not end for a network drop. The probe must be a read + /// (mutating nothing, moving no client size) and must resolve through the same + /// `%begin`/`%end` correlation as any other command rather than a bespoke heartbeat. + @MainActor @Test func alivenessIsProvedByAControlModeRoundTrip() { + let connection = RemoteTmuxControlConnection( + host: RemoteTmuxHost(destination: "user@host", transport: .et), + sessionName: "work" + ) + // Never started, so there is no stream to carry a question: the caller is told the + // probe did not leave, rather than being left waiting on a completion that cannot come. + var completionFired = false + let enqueued = connection.probeLiveness { _ in completionFired = true } + #expect(!enqueued, "a probe must report that it could not be sent on a dead stream") + #expect(!completionFired, "no completion may fire for a probe that never left") + } + + /// The property that makes the probe necessary in the first place. + @Test func aStallIsNotADeathForATransportThatReconnectsItself() { + let et = RemoteTmuxETTransportProfile() + #expect(et.reconnectsInternally) + // EOF from such a transport means the session is genuinely over... + #expect(RemoteTmuxStreamEndDisposition.forStreamEnd( + reconnectsInternally: true) == .sessionOver) + // ...whereas ssh's EOF is cmux's cue to respawn. Same event, opposite meaning, which + // is why a stall has to be diagnosed by asking rather than by waiting for EOF. + #expect(RemoteTmuxStreamEndDisposition.forStreamEnd( + reconnectsInternally: false) == .reconnect) + } + + // MARK: - Runtime selection + + /// A host carries its transport, so selection reaches the spawn without a global switch. + @Test func aHostSelectsItsOwnTransport() { + let sshHost = RemoteTmuxHost(destination: "user@host") + #expect(sshHost.transport == .ssh, "an unspecified host must behave exactly as before") + #expect(!sshHost.transport.profile(port: nil).requiresPseudoTerminal) + + let etHost = RemoteTmuxHost(destination: "user@host", transport: .et) + let profile = etHost.transport.profile(port: etHost.port) + #expect(profile.requiresPseudoTerminal) + #expect(profile.reconnectsInternally) + } + + /// The transport's port is NOT ssh's port, and conflating them breaks every one-shot. + /// + /// One-shot discovery and mutation keep riding ssh even when the control stream does + /// not, so a single `port` field pointed ssh at etserver's port and every one-shot died + /// with `kex_exchange_identification: Connection reset`. Found by an end-to-end run, + /// not by argv inspection — which is exactly why this test exists. + @Test func theTransportPortIsSeparateFromTheSSHPort() { + // ssh keeps 22 for its one-shots while et carries the stream on 2039. + let host = RemoteTmuxHost( + destination: "user@host", port: 22, transport: .et, transportPort: 2039 + ) + let streamArgv = host.transport.profile(port: host.transportPort) + .controlStreamArgv(host: host, sessionName: "work", createIfMissing: false) + #expect(consecutive(streamArgv, "-p", "2039"), "the control stream must use et's port") + + let oneShot = RemoteTmuxSSHTransportProfile().oneShotArgv(host: host, remoteCommand: "true") + #expect(!consecutive(oneShot, "-p", "2039"), "a one-shot must never be sent to et's port") + + // Unset means etserver's documented default, never ssh's 22. + let defaulted = RemoteTmuxHost(destination: "user@host", transport: .et) + let defaultArgv = defaulted.transport.profile(port: defaulted.transportPort) + .controlStreamArgv(host: defaulted, sessionName: "work", createIfMissing: false) + #expect(consecutive(defaultArgv, "-p", "2022")) + } + + /// An unrecognized transport is refused rather than becoming an unspawnable host. + @Test func anUnknownTransportIsRejected() { + #expect(RemoteTmuxTransportKind.parse("ssh") == .ssh) + #expect(RemoteTmuxTransportKind.parse("et") == .et) + #expect(RemoteTmuxTransportKind.parse("ET") == .et) + #expect(RemoteTmuxTransportKind.parse(nil) == .ssh, "unset means today's behavior") + #expect(RemoteTmuxTransportKind.parse("") == .ssh) + #expect(RemoteTmuxTransportKind.parse("mosh") == nil) + #expect(RemoteTmuxTransportKind.parse("../../bin/sh") == nil) + } + + /// A connection with no explicit profile takes the host's, so nothing has to remember + /// to pass it at each construction site. + @MainActor @Test func aConnectionTakesItsProfileFromTheHost() { + let et = RemoteTmuxControlConnection( + host: RemoteTmuxHost(destination: "user@host", transport: .et), + sessionName: "work" + ) + #expect(et.transportProfile.requiresPseudoTerminal) + + let ssh = RemoteTmuxControlConnection( + host: RemoteTmuxHost(destination: "user@host"), + sessionName: "work" + ) + #expect(!ssh.transportProfile.requiresPseudoTerminal) + } + + private func consecutive(_ args: [String], _ a: String, _ b: String) -> Bool { + for index in args.indices.dropLast() where args[index] == a && args[index + 1] == b { + return true + } + return false + } +} + +// MARK: - Layer 3: seeded model-based fuzz over transport behavior + +/// Deterministic seedable RNG (SplitMix64), matching the repo's other fuzz suites: every +/// failure reproduces from the seed plus step index printed in the assertion. +private struct SplitMix64 { + private var state: UInt64 + init(seed: UInt64) { state = seed } + mutating func next() -> UInt64 { + state &+= 0x9E37_79B9_7F4A_7C15 + var z = state + z = (z ^ (z >> 30)) &* 0xBF58_476D_1CE4_E5B9 + z = (z ^ (z >> 27)) &* 0x94D0_49BB_1331_11EB + return z ^ (z >> 31) + } + mutating func int(_ range: Range) -> Int { + range.lowerBound + Int(next() % UInt64(range.count)) + } +} + +/// Fuzzes the decisions a transport seam makes, against a reference model of the transport +/// and the remote server. +/// +/// The point is invariant 1 below: cmux must never respawn a transport that owns its own +/// reconnection. ssh-era code reacted to stdout EOF, and a transport that pauses and resumes +/// instead of ending would have its session thrown away by that reflex. The model carries +/// the three pieces of state the ssh-only world never had — is the process alive, is the +/// stream flowing, does the remote session still exist. +@Suite struct RemoteTmuxTransportFuzzTests { + + /// Fixed seeds. This list only ever grows: when a seed finds a bug, the fix lands and the + /// seed stays, so the same case can never regress silently. + static let seeds: [UInt64] = [ + 0x5EED_0001, 0x5EED_0002, 0x5EED_0003, 0x5EED_0004, + 0xA11C_E5, 0xB0B_CAFE, 0xDEAD_BEEF, 0x1234_5678, + ] + + /// What the transport can do to the stream. + private enum Event: CaseIterable { + case stall // network pause: no bytes, process alive + case resume // bytes flow again + case dropMidFrameThenResume + case exitClean // the command finished: session over + case exitAuthFailure + case exitTransientFailure + case killSessionRemotely + } + + /// The reference model: what a correct cmux would do. + private struct Model { + var processAlive = true + var streamFlowing = true + var sessionExists = true + var spawnCount = 1 + var ended = false + var authOutcomes = 0 + var retryScheduled = 0 + } + + @Test(arguments: seeds) + func aTransportThatOwnsReconnectionIsNeverRespawnedForAStall(seed: UInt64) { + var rng = SplitMix64(seed: seed) + let profile = RemoteTmuxETTransportProfile() + #expect(profile.reconnectsInternally, "this suite is about internally-reconnecting transports") + + var model = Model() + let spawnsAtStart = model.spawnCount + + for step in 0..<40 { + let event = Event.allCases[rng.int(0.. [String] + /// argv for a one-shot remote command (discovery, mutations). + func oneShotArgv(host: RemoteTmuxHost, remoteCommand: String) -> [String] + /// Whether the transport recovers from network loss on its own, or whether cmux + /// must respawn it. + var reconnectsInternally: Bool { get } +} +``` + +`SSHTransport` is the current behavior, unchanged and still the default. A second +implementation wraps a persistent-session transport. For ET that is: + +``` +et --command 'exec -CC attach-session -t ' [--port N] @ +``` + +Four details are load-bearing for any such transport: + +- **Run the command, do not open a shell.** ET's `--command` runs one command and exits. + Prefix it with `exec` so the remote process tree does not keep a shell parent around. +- **The remote command still needs the tmux resolver.** A non-login remote shell often has + a minimal `PATH`, which is why `RemoteTmuxHost.tmuxRemoteCommand` wraps `tmux` in a + resolver script. That is a property of the remote shell, not of ssh, so it applies here + identically. +- **The port is per-transport and per-host.** Keep it configurable instead of assuming + ssh's 22. +- **Never pass a flag that kills other sessions.** ET's `-x` / `--kill-other-sessions` + terminates every one of that user's sessions on the host, not just stale ones. Some + wrappers pass it by default, so an unrelated reconnect elsewhere can kill cmux's + session; recover by recreating on unexpected exit rather than trying to hold a claim. + +One-shot commands can keep using ssh over the existing ControlMaster even when the control +stream rides another transport. `RemoteTmuxSSHTransport.ensureMasterReady()` already +funnels the cold-start burst through a single master open, and that logic is independent +of how the `-CC` stream is carried. + +## Seam 2: reconnect stops being EOF-driven + +This is the part that changes behavior, and the reason `reconnectsInternally` exists +rather than just swapping argv. + +Today `handleStreamEnd` treats stdout EOF as the signal to reconnect and +`beginReconnecting()` respawns with backoff. A transport that reconnects internally +produces no EOF for a network drop: the stream pauses and resumes. Reacting only to EOF is +therefore correct and cmux notices nothing, which is the goal. What it must not do is +treat a *stall* as death and respawn on a timer, because that discards a session the +transport was about to recover. + +So the failure mode moves from "stream ended" to "stream is alive but wedged", and a +transport that reconnects internally needs a liveness check instead: + +1. the transport process is still alive, and +2. a control-mode round-trip completes. + +cmux already has the round-trip primitive: a bounded `display-message -p ok` query, used as +`awaitCommandBarrier` in `RemoteTmuxViewConnection`. Reuse it rather than inventing a +heartbeat. A genuine transport exit still means the session is over and routes to the +existing session-gone path. + +There is a bug here worth fixing independently of any new transport. +`handleStreamEnd`'s classifier only distinguishes "session gone" (end) from everything else +(retry with backoff, forever). A reconnect that fails because the host wants interactive +authentication lands in the second bucket, so it retries silently and forever. cmux already +detects that case (`RemoteTmuxSSHTransport.indicatesAuthRequired`) and already has a path +for handing the user an interactive ssh on initial attach +(`RemoteTmuxAttachOutcome.authRequired(sshArgv:)`, built by +`RemoteTmuxHost.interactiveAuthInvocation()`). The reconnect path should reuse that outcome +instead of looping. + +## Seam 3: a pre-connect hook + +Some hosts need a step before the connection that cmux has no business knowing about: +minting a short-lived credential, unlocking an agent, refreshing a token. Rather than +teaching cmux any of them, give a host an optional command, in the same spirit as the +existing per-host upload command: + +``` +preseedCommand: # run as ` ` before opening a connection +``` + +cmux runs it before opening or reopening a connection, treats a non-zero exit as non-fatal +(proceed and let the connection fail normally, so a broken hook cannot make a host +unreachable), and ignores its output. Unset means today's behavior. + +Two constraints, both learned the hard way wiring one up: + +- **Run it once per connection, not once per config parse.** Anything minting a single-use + credential must not run concurrently with itself; two mints racing can invalidate each + other. The right home is the same single-flight path that opens the shared master + (`ensureMasterReady()`), not a per-command hook. +- **The connection that follows must allow interactive auth methods.** A pre-connect step + that satisfies a second factor typically does so through a keyboard-interactive exchange + that completes without prompting. Opening that connection with `BatchMode=yes` refuses + the method outright and fails `Permission denied (keyboard-interactive)` even though + nothing would have prompted. Piped stdin already makes a genuine prompt fail fast, so + batch mode is not what protects against hanging. + +## Proving it + +Scope this at the transport, not the mirror. Render fidelity already has strong coverage +that a transport change does not improve on: `remote-tmux-render-harness.sh` asserts the +mirror's visible screen equals the remote pane's visible screen across attach, resize, +rapid re-attach and reconnect, and `remote-tmux-shape-zoo.sh` plus +`RemoteTmuxSizingUITests` cover geometry. Those stay as the regression gate and should go +green unchanged; if a transport swap breaks them, that is the answer, and re-testing +rendering under a second transport adds cost without signal. + +What is genuinely uncovered is the **connection state machine**: what cmux does when a +stream stalls, resumes, dies, or fails to authenticate, and who resolves the commands that +were in flight when it happened. Every layer below targets that. + +Three layers, cheapest first. The mock keeps unit tests fast, the real servers keep the +mock honest, and the fuzz is what finds the state-machine bugs. + +### Layer 1: mock transport (unit speed, no network) + +`scripts/remote-tmux-e2e-ssh-shim.sh` already does this for ssh: it strips the option +framing, runs the "remote" command locally, and allocates a pty with `script(1)` only when +`-t`/`-tt` asked for one. `RemoteTmuxHost.defaultSSHExecutablePath()` honors +`CMUX_REMOTE_TMUX_SSH_FOR_TESTING` in DEBUG so the real app process can be pointed at it. + +Add the same seam for the transport binary (an ET shim honoring `--command` instead of +ssh's framing) so `ETTransport` gets identical treatment. The shim must reproduce the two +behaviors that bite: hand the remote command to a shell that re-splits it, and allocate a +pty for the control stream but **not** for one-shot probes, because probe classification +reads stderr and a pty merges stderr into the stream. + +It must also be able to *simulate the interesting failures on demand*, which the ssh shim +has no reason to do: pause the stream without exiting (the wedge), resume after a pause +(internal reconnect), drop mid-frame, and exit with a chosen status. Drive those from +environment variables so a unit test can request one deterministically. + +### Layer 2: real sshd and a real ET server (end-to-end truth) + +**No containers.** Nothing in remote-tmux uses Docker today, and this should not introduce +it. (`tests/fixtures/ssh-remote/` has a Dockerfile, but that fixture belongs to the +cloud-VM image builder, not to remote-tmux.) The established pattern is a **loopback ssh +alias whose forced command pins an isolated `TMUX_TMPDIR`**, so a harness drives real ssh +against the machine's own sshd and can never touch the developer's real tmux: + +``` +Host cmux-srvA + HostName 127.0.0.1 + RemoteCommand TMUX_TMPDIR=/tmp/cmux-srvA $SHELL -l + RequestTTY yes +``` + +`scripts/remote-tmux-render-harness.sh` uses exactly that and returns the number of failed +scenarios as its exit code. `scripts/remote-tmux-shape-zoo.sh` builds a geometry zoo on a +real host over one ordinary ssh connection for manual exercise. Reuse both conventions: +loopback, isolated tmpdir, generated keys on a nonstandard port, exit code = failed +scenario count. + +ET drops into that pattern without a container, because an ET client bootstraps over ssh to +the host and then speaks to an `etserver` there. Pointed at loopback, the whole path is +local and real: + +``` +cmux → et client (upstream) → ssh 127.0.0.1 (real sshd) → etserver/etterminal → tmux -CC +``` + +Requirements for it to be trustworthy: + +- **Use upstream ET, pinned to a tag** (Homebrew formula or a source build), so the test + proves compatibility with public ET rather than with a fork. +- **Isolate hard**: a dedicated `TMUX_TMPDIR`, a nonstandard etserver port, per-run + generated keys with explicit `IdentityFile`/`IdentitiesOnly`. Never the default tmux + socket. +- **Exercise a real drop, and sever it at the network layer.** This is the one property only + a real transport can prove. Run the client through a small TCP relay the harness can pause + and resume (a packet-filter rule works too, but a relay needs no privileges and is + deterministic). Killing the `et` process tests the wrong thing: that is the session-gone + path, not a recoverable outage. +- **Assert on cmux's observable state, not log text**: the mirror stays populated across the + outage, the spawn counter does not increment, and a command issued after the resume + completes. + +Skip cleanly with a stated reason when `et`/`etserver` is absent, rather than passing +vacuously. + +### Layer 3: seeded model-based fuzz (where the bugs are) + +Follow `RemoteTmuxMultiplexFuzzTests` exactly: `SplitMix64` seeded from the test argument +so every draw is deterministic, a fixed seed list under `@Test(arguments:)`, a tiny +reference model playing the transport and the remote server, and a step loop that mutates +the model, drives one action, then checks the full invariant set. Failure messages carry +seed plus step index so a case reproduces exactly. No `Date()`, no +`SystemRandomNumberGenerator`, no wall-clock. + +The model needs three pieces of state the ssh-only world did not have: whether the +transport process is alive, whether the stream is flowing or stalled, and whether the +remote session still exists. + +Step actions to draw from: + +| Category | Actions | +|---|---| +| Transport | stall the stream; resume after a stall; drop mid-frame then resume; exit cleanly; exit with an auth failure; exit with a transient failure; refuse to launch (binary missing) | +| Session | kill the session remotely during an outage; create a session; rename; churn windows | +| cmux | issue a one-shot; issue a command batch; resize; stop the connection; close the window; quit | +| Hook | hook succeeds; hook exits non-zero; hook hangs past its timeout; two opens race the hook | + +Invariants to assert after every step: + +1. **No respawn while an internally-reconnecting transport is alive.** The spawn counter + must not increment for a stall or a resume. This is the property the whole design rests + on, and the easiest one to regress. +2. **A stall never ends the connection**, and a genuine transport exit always does. +3. **An auth-required failure always surfaces an auth outcome** and never enters an + unbounded retry loop. +4. **Every pending command resolves exactly once**, either completing or failing. Nothing + is silently dropped across a stall, a resume, or a teardown. +5. **The parser never desyncs.** A frame split across a resume boundary either completes or + is discarded whole; a partial frame must never be delivered as a message. +6. **The pre-connect hook runs at most once per connection open**, even when opens race. +7. **A session the user killed is never silently recreated** (the existing sticky + never-surfaced-a-workspace gate must still hold under transport churn). +8. **Teardown is ordering-independent**: a resume that lands after `stop()` changes + nothing. + +Ratchet discipline, which is the part that makes this worth doing: when a seed fails, fix +the product, then **add that seed to the permanent list** rather than only fixing the bug. +When a real-world failure shows up that no seed produced, add the step action that would +have produced it and re-run the whole list. The suite only ever grows. A fuzz suite that +stays at its original eight seeds after finding bugs is not ratcheting, it is decoration. + +## Notes on carrying a control protocol over a PTY transport + +`tmux -CC` is line-oriented, and any transport that allocates a remote PTY translates `\n` +to `\r\n` on the way back. cmux already runs `ssh -tt` (also a remote PTY) so +`RemoteTmuxControlStreamParser` tolerates `\r` today. Keep an explicit test for it rather +than rediscovering it after a transport swap. + +## Order of work + +1. Fix the reconnect classifier so an auth-required reconnect surfaces the interactive path + instead of retrying forever. Independent of any new transport and testable red-first: a + reconnect whose transport fails with a permission error must produce an auth outcome, + not a backoff loop. +2. Introduce `RemoteTmuxTransport` with `SSHTransport` as the only implementation. No + behavior change, so the existing suites are the regression gate. +3. Add the fuzz harness against the mock transport, with the invariants above. It should + pass for `SSHTransport` before any new transport exists. +4. Add the persistent-session transport behind per-host opt-in, with the liveness check and + a fallback to `SSHTransport` when the transport binary is missing. Turn on the + internally-reconnecting arm of the fuzz. +5. Add the real sshd plus upstream-ET end-to-end fixture, including a real network drop. +6. Add the pre-connect hook, with the race and timeout cases in the fuzz. diff --git a/scripts/remote-tmux-et-e2e.sh b/scripts/remote-tmux-et-e2e.sh new file mode 100755 index 000000000000..d787f39762b0 --- /dev/null +++ b/scripts/remote-tmux-et-e2e.sh @@ -0,0 +1,84 @@ +#!/bin/bash +# ============================================================================ +# End-to-end: cmux carries a real tmux control stream over a real EternalTerminal. +# +# Everything else about the seam is argv shape and decision rules, which unit tests +# can pin. This is the only check that the app actually drives ET: the pty spawn, +# the ~1.2 KB of login-shell preamble before `%begin`, the CRLF line endings, and +# the control protocol surviving all of it inside the running app. +# +# PREREQUISITES +# - scripts/remote-tmux-et-host.sh has brought up a loopback etserver. +# - An ssh alias for that host (one-shot discovery still rides ssh by design). +# - A tagged Debug app running with remote-tmux beta on; pass CMUX_TAG=. +# +# Exit code is the number of failed checks (0 = all green). +# ============================================================================ +set -uo pipefail + +TAG="${CMUX_TAG:?set CMUX_TAG= of a running tagged Debug app}" +HOST="${CMUX_ET_HOST:-cmux-ethost}" +PORT="${CMUX_ET_PORT:-2039}" +SESSION="${CMUX_ET_SESSION:-etmirror}" +CLI=(scripts/cmux-debug-cli.sh) +FAILURES=0 + +log() { printf '%s %s\n' "$(date '+%H:%M:%S')" "$*"; } +pass() { printf ' ✅ %s\n' "$*"; } +fail() { printf ' ❌ %s\n' "$*"; FAILURES=$((FAILURES + 1)); } +cli() { CMUX_QUIET=1 CMUX_TAG="$TAG" "${CLI[@]}" "$@" 2>&1; } + +await() { + local what="$1" timeout="$2"; shift 2 + local deadline=$((SECONDS + timeout)) + while [ "$SECONDS" -lt "$deadline" ]; do + if "$@"; then return 0; fi + sleep 2 + done + log "timed out after ${timeout}s waiting for: $what" + return 1 +} + +cd "$(dirname "$0")/.." || exit 1 + +app_ready() { cli list-workspaces >/dev/null 2>&1; } +if ! await "the app's control socket" 90 app_ready; then + log "no response on the tagged app socket for CMUX_TAG=$TAG" + exit 1 +fi + +command -v et >/dev/null 2>&1 || { log "et is not installed"; exit 1; } +nc -z 127.0.0.1 "$PORT" 2>/dev/null || { log "no etserver on 127.0.0.1:$PORT"; exit 1; } + +log "=== attaching session '$SESSION' over the et transport" +# A named attach rather than discovery: this proves the control stream, without +# mirroring whatever else happens to be on the host's default tmux server. +# `cmux rpc` is the raw v2 call; a named attach avoids mirroring whatever else happens to +# be on the host's default tmux server. +RESULT="$(cli rpc remote.tmux.attach \ + "{\"host\":\"$HOST\",\"session\":\"$SESSION\",\"port\":$PORT,\"transport\":\"et\"}" 2>&1)" +log "attach result: $(printf '%s' "$RESULT" | head -c 300)" + +# The decisive evidence is a live et process carrying tmux for this host, spawned by +# the app rather than by this script. +et_stream_live() { + pgrep -f "attach-session" 2>/dev/null | while read -r p; do + ps -o command= -p "$p" 2>/dev/null | grep -q "et " && echo x + done | grep -q x +} +if await "an et-carried control stream spawned by cmux" 60 et_stream_live; then + pass "cmux spawned a control stream over et" +else + fail "no et-carried control stream appeared" +fi + +# And it must be under a pty: a bare pipe spawn is exactly what produces no output. +pty_wrapped() { pgrep -fl "script -q /dev/null" >/dev/null 2>&1; } +if pty_wrapped; then + pass "the transport was spawned under a pseudo-terminal" +else + fail "no pty wrapper around the transport (et emits nothing on pipes)" +fi + +log "=== $FAILURES failed check(s)" +exit "$FAILURES" diff --git a/scripts/remote-tmux-et-host.sh b/scripts/remote-tmux-et-host.sh new file mode 100755 index 000000000000..a57062bd8152 --- /dev/null +++ b/scripts/remote-tmux-et-host.sh @@ -0,0 +1,66 @@ +#!/bin/bash +# ============================================================================ +# Brings up a REAL EternalTerminal server on loopback, so the transport seam can +# be exercised against `et` itself rather than a mock. +# +# ET is not ssh: it terminates the connection on the server side and reconnects +# internally after a network change, which is the whole reason the seam exists. +# The only way to know cmux's argv and its reconnect ownership are right is to +# carry a real `tmux -CC` control stream over a real et. +# +# Usage: scripts/remote-tmux-et-host.sh [name] [port] +# Exit 0 with the connection details on stdout, non-zero if it could not start. +# ============================================================================ +set -uo pipefail + +NAME="${1:-cmux-ethost}" +PORT="${2:-2039}" + +command -v et >/dev/null 2>&1 || { echo "et not installed" >&2; exit 2; } +command -v etserver >/dev/null 2>&1 || { echo "etserver not installed" >&2; exit 2; } + +STATE_ROOT="$HOME/Library/Caches/cmux/remote-tmux-et" +DIR="$STATE_ROOT/$NAME" +if [ -L "$STATE_ROOT" ] || [ -L "$DIR" ]; then + echo "refusing symlinked et state path" >&2; exit 1 +fi +umask 077 +mkdir -p "$DIR/logs" "$DIR/tmux" +chmod 700 "$DIR" + +# etserver wants a pidfile it can write; /var/run needs root, so keep it local. +PIDFILE="$DIR/etserver.pid" + +if [ -f "$PIDFILE" ] && kill -0 "$(cat "$PIDFILE" 2>/dev/null)" 2>/dev/null; then + echo "already running (pid $(cat "$PIDFILE"))" +else + # Bind loopback only. This is a test server on a developer machine, and an + # et server accepts real shells — it must not be reachable off-box. + etserver --port "$PORT" --bindip 127.0.0.1 --pidfile "$PIDFILE" \ + --logdir "$DIR/logs" --daemon >"$DIR/logs/start.out" 2>&1 + rc=$? + if [ "$rc" -ne 0 ]; then + echo "etserver failed to start (rc=$rc):" >&2 + tail -5 "$DIR/logs/start.out" >&2 + exit "$rc" + fi +fi + +# Wait for the port rather than sleeping: startup is fast but not instant. +for _ in $(seq 1 30); do + if nc -z 127.0.0.1 "$PORT" 2>/dev/null; then break; fi + sleep 0.5 +done +if ! nc -z 127.0.0.1 "$PORT" 2>/dev/null; then + echo "etserver is not accepting connections on $PORT" >&2 + tail -20 "$DIR/logs"/* 2>/dev/null >&2 + exit 1 +fi + +cat <' $USER@127.0.0.1 +INFO From 1e9a0ac67aee762599dc00ba660a43271fda5487 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 09:30:55 -0700 Subject: [PATCH 02/23] remote-tmux: make the et transport actually carry a control stream MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The et transport could never carry `tmux -CC`. Two separate faults, both found by running the end-to-end harness against a real etserver rather than by reading argv. et does not exec its `--command`: it types it into a login shell and appends `; exit`. That shell reads from a pty in canonical mode, which delivers at most MAX_CANON (1024 on macOS) bytes per line, and the PATH resolver ssh needs is about 1113 bytes. The line never completed, so the shell ran nothing and the stream sat silent until the attach timed out with nothing to explain it. Measured: a 23-byte command exits 0, a 1123-byte one hangs. et gets plain `tmux` now, which is correct as well as short, because a login shell already has the user's full PATH. ssh keeps the resolver — a non-login shell with a minimal PATH is the case it exists for. With bytes flowing, the stream still never reached control mode. The parser recognised tmux's `ESC P 1000 p` only at the start of a line, and the login shell leaves its echo and OSC title sequences ahead of it with no newline between, so `.enter` never fired while every later notification parsed normally. Commands are withheld until `.enter`, so the mirror waited forever. The scan now finds the sequence anywhere in the line and drops the shell noise before it, scoped to before control mode is entered and outside a command block: block content is raw `capture-pane -e` bytes that can carry that DCS legitimately, and matching it there would cut the pane apart. Tests. The fixture test for the real et stream asserted only that `%session-changed` parsed, which is true on the broken code; it now asserts `.enter` is produced and precedes it. The argv test asserted the resolver was present, pinning the bug, and now pins its absence. New: a bound on the command against MAX_CANON including a long session name, injection-safe quoting of the session name now that the resolver no longer does it, mid-line enter recognition, and a DCS inside block content staying whole. The harness proved neither fault, so it grew the checks that would: it asserted a process listing, which both faults satisfied, and skipped the session it attaches by name. It now provisions that session on the server both transports reach — one-shot commands ride ssh while the stream rides et, so a private TMUX_TMPDIR hides it from the ssh side — refuses to adopt a session it did not create, and reads cmux's own view of the stream: the handshake parsed and windows arrived. --- Sources/RemoteTmuxControlStreamParser.swift | 39 ++++++++++- Sources/RemoteTmuxTransportRegistry.swift | 27 ++++--- cmuxTests/RemoteTmuxControlParserTests.swift | 35 ++++++++++ .../RemoteTmuxProxyTransportRetryTests.swift | 70 ++++++++++++++++++- docs/remote-tmux-transport-seam.md | 13 ++-- scripts/remote-tmux-et-e2e.sh | 54 +++++++++----- scripts/remote-tmux-et-host.sh | 23 ++++++ 7 files changed, 225 insertions(+), 36 deletions(-) diff --git a/Sources/RemoteTmuxControlStreamParser.swift b/Sources/RemoteTmuxControlStreamParser.swift index 7a4be61958de..96d78b02cfe0 100644 --- a/Sources/RemoteTmuxControlStreamParser.swift +++ b/Sources/RemoteTmuxControlStreamParser.swift @@ -21,6 +21,10 @@ struct RemoteTmuxControlStreamParser { private let maxCommandBlockBytes: Int private var buffer: [UInt8] = [] private var inBlock = false + /// Whether control mode has been entered. Entering happens once per stream, so the scan for + /// the enter DCS stops after it — searching later lines could match a DCS that is genuinely + /// part of a pane's own bytes. + private var sawEnter = false private var blockNumber = 0 private var blockLines: [String] = [] private var blockBufferedBytes = 0 @@ -36,6 +40,22 @@ struct RemoteTmuxControlStreamParser { /// The DCS sequence tmux emits to enter control mode: `ESC P 1000 p`. private static let enterSequence: [UInt8] = [0x1b, 0x50, 0x31, 0x30, 0x30, 0x30, 0x70] + /// Index range of the first occurrence of `needle` in `haystack`, or nil. + private static func firstRange(of needle: [UInt8], in haystack: [UInt8]) -> Range? { + guard !needle.isEmpty, haystack.count >= needle.count else { return nil } + let last = haystack.count - needle.count + var start = 0 + while start <= last { + if haystack[start] == needle[0] { + var offset = 1 + while offset < needle.count, haystack[start + offset] == needle[offset] { offset += 1 } + if offset == needle.count { return start..<(start + needle.count) } + } + start += 1 + } + return nil + } + /// ASCII bytes of the `%output ` notification prefix (used to detect and parse /// `%output` lines from raw bytes, before any String decode). private static let outputPrefix: [UInt8] = Array("%output ".utf8) @@ -67,10 +87,23 @@ struct RemoteTmuxControlStreamParser { var bytes = rawBytes var prefixMessages: [RemoteTmuxControlMessage] = [] - // Strip a leading enter DCS (it is prepended to the first %begin line). - if bytes.starts(with: Self.enterSequence) { + // Strip the enter DCS. It usually opens the line, but it does not have to: a transport + // that types the command into a login shell (et) leaves the shell's echo and its OSC + // title sequences on the same line, with the DCS arriving partway through and no + // newline before it. Requiring it at offset 0 means `.enter` is never emitted there, + // and since commands are withheld until `.enter`, the mirror waits forever while the + // notifications after it parse normally. + // + // Everything before the DCS is pre-control-mode shell noise by definition — tmux emits + // this sequence when it takes over the terminal — so dropping that prefix is right + // rather than merely convenient. + // Scoped to before control mode is entered and outside a command block: block content is + // raw pane bytes (`capture-pane -e`) that can legitimately contain this DCS, and matching + // it there would cut the painted pane apart — the same hazard the ST strip below avoids. + if !sawEnter, !inBlock, let dcs = Self.firstRange(of: Self.enterSequence, in: bytes) { + sawEnter = true prefixMessages.append(.enter) - bytes.removeFirst(Self.enterSequence.count) + bytes.removeFirst(dcs.upperBound) } // Drop ST (ESC \) DCS-teardown framing — but ONLY on notification lines. // Command-block content (e.g. `capture-pane -e` output) is raw terminal diff --git a/Sources/RemoteTmuxTransportRegistry.swift b/Sources/RemoteTmuxTransportRegistry.swift index bda7696c4523..aa7281c1eba9 100644 --- a/Sources/RemoteTmuxTransportRegistry.swift +++ b/Sources/RemoteTmuxTransportRegistry.swift @@ -239,15 +239,24 @@ struct RemoteTmuxETTransportProfile: RemoteTmuxTransportProfile { sessionName: String, createIfMissing: Bool ) -> [String] { - // The resolver is still required: a non-login remote shell has a minimal PATH, and - // that is a property of the remote shell rather than of ssh. - let remote = RemoteTmuxHost.tmuxRemoteCommand( - arguments: [ - "-CC", - createIfMissing ? "new-session" : "attach-session", - "-t", sessionName, - ] - ) + // Plain `tmux`, not the PATH resolver ssh needs, and the reason is a hard limit rather + // than a preference. `et` does not exec the command: it types it into a login shell and + // appends `; exit`. That shell reads from a pty in canonical mode, which delivers at + // most MAX_CANON (1024 on macOS) bytes per line, and the resolver is ~1113 bytes. The + // line never completes, so the shell runs nothing and the stream sits silent until the + // attach times out — measured against et 6.2.11. + // + // Dropping the resolver is safe precisely because it is a login shell: it has the user's + // full PATH, so it finds tmux itself. ssh is the opposite case, running a non-login shell + // with a minimal PATH, which is why that profile still needs the resolver. + let remote = ([ + "tmux", + "-CC", + createIfMissing ? "new-session" : "attach-session", + "-t", sessionName, + ] as [String]) + .map(RemoteTmuxHost.shellSingleQuoted) + .joined(separator: " ") // Arguments only: the executable is supplied separately (see ``executablePath()``), // exactly as the ssh profile does. Including it here would pass `et` twice. var argv = ["-p", String(port)] diff --git a/cmuxTests/RemoteTmuxControlParserTests.swift b/cmuxTests/RemoteTmuxControlParserTests.swift index 7130b489c437..8527048983ec 100644 --- a/cmuxTests/RemoteTmuxControlParserTests.swift +++ b/cmuxTests/RemoteTmuxControlParserTests.swift @@ -416,6 +416,40 @@ import Testing // Trailing junk after a valid node fails (cursor must reach the end). #expect(RemoteTmuxRawLayoutParser.parse("80x24,0,0,1xyz") == nil) } + + /// The enter DCS is recognised partway through a line, because a transport that types the + /// command into a login shell (et) leaves that shell's echo and OSC title sequences ahead of + /// it with no newline between. Requiring offset 0 meant `.enter` never fired on a real et + /// stream, and commands are withheld until `.enter`. + @Test func enterIsFoundWhenShellEchoPrecedesTheDCS() { + let messages = parse("\u{1B}]0;ejc3@host\u{07}exec tmux -CC attach\u{1B}P1000p%begin 1 1 0\r\n") + #expect(messages.contains { if case .enter = $0 { return true }; return false }) + } + + /// But only before control mode is entered, and never inside a command block: block content is + /// raw pane bytes from `capture-pane -e`, which can contain this DCS legitimately. Treating + /// that as a second enter would also cut the captured pane apart at the match. + @Test func aDCSInsideBlockContentIsNotASecondEnter() { + let enter = "\u{1b}P1000p" + let messages = parse( + enter + "%begin 1700000000 1 0\r\n" + + "ok\r\n" + + "%end 1700000000 1 0\r\n" + + "%begin 1700000000 2 0\r\n" + + "pane painted \u{1b}P1000p still the same pane\r\n" + + "%end 1700000000 2 0\r\n" + ) + let enters = messages.filter { if case .enter = $0 { return true }; return false } + #expect(enters.count == 1, "expected exactly one .enter, saw \(enters.count)") + // The captured pane's bytes survive whole rather than being cut at the embedded DCS. + #expect(messages.contains( + .commandResult( + commandNumber: 2, + lines: ["pane painted \u{1b}P1000p still the same pane"], + isError: false + ) + )) + } } /// Behavior tests for the per-pane foreground classification @@ -549,4 +583,5 @@ import Testing #expect(RemoteTmuxControlConnection.parseActivityQueryLine("garbage") == nil) #expect(RemoteTmuxControlConnection.parseActivityQueryLine("") == nil) } + } diff --git a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift index 8a66331e4903..915bb90dabea 100644 --- a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift +++ b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift @@ -289,6 +289,25 @@ import Testing return false } #expect(sawSessionChange, "expected %session-changed to parse after the ET preamble") + + // `.enter` is the message that matters, and asserting only on %session-changed hid a + // real bug: the login shell's echoed title sequences land on the same line as the + // enter DCS, so a parser that requires the DCS at the start of a line never emits + // `.enter`. Everything downstream waits on it — the connection stays out of + // `.connected` and the attach dies on a timeout while later notifications parse + // normally, which is exactly what a real et host did. + let sawEnter = messages.contains { message in + if case .enter = message { return true } + return false + } + #expect(sawEnter, "expected .enter from the ET stream's mid-line enter DCS") + // Order matters too: commands are withheld until `.enter`, so it has to arrive before + // the notifications that follow it in the stream. + let enterIndex = messages.firstIndex { if case .enter = $0 { return true }; return false } + let changeIndex = messages.firstIndex { if case .sessionChanged = $0 { return true }; return false } + if let enterIndex, let changeIndex { + #expect(enterIndex < changeIndex, "`.enter` must precede %session-changed") + } } // MARK: - The profile's argv, measured against the real client @@ -307,14 +326,59 @@ import Testing #expect(argv.last == "user@127.0.0.1") let command = argv.first(where: { $0.hasPrefix("exec ") }) #expect(command?.contains("attach-session") == true) - // The resolver is still needed: a minimal remote PATH is a property of the remote - // shell, not of ssh. - #expect(command?.contains("cmux-remote-executable") == true) + // Plain tmux, not ssh's PATH resolver. et types the command into a login shell, so the + // shell resolves tmux from the user's own PATH — and the resolver would not fit anyway + // (see etCommandFitsWhatALoginShellCanRead). + #expect(command?.contains("cmux-remote-executable") == false) + #expect(command?.contains("'tmux'") == true) // Never kill the user's other sessions on that host. #expect(!argv.contains("-x")) #expect(!argv.contains("--kill-other-sessions")) } + /// et types the command into a login shell instead of exec'ing it, and that shell reads from + /// a pty in canonical mode. macOS delivers at most MAX_CANON (1024) bytes per line, so a + /// longer command never completes a line: the shell runs nothing, et emits nothing, and the + /// attach dies on a timeout with no error to explain it. Measured against et 6.2.11, where + /// ssh's ~1113-byte PATH resolver hung and a 40-byte plain `tmux` produced `%begin`. + /// + /// Long session names are the realistic way to cross the line, so the bound is checked + /// against one rather than only the short names the other tests use. + @Test func etCommandFitsWhatALoginShellCanRead() { + let maxCanon = 1024 + for session in ["s", "work session", String(repeating: "session-", count: 24)] { + let argv = RemoteTmuxETTransportProfile(port: 2039).controlStreamArgv( + host: RemoteTmuxHost(destination: "user@host"), + sessionName: session, + createIfMissing: false + ) + let command = try? #require(argv.first(where: { $0.hasPrefix("exec ") })) + let byteCount = (command ?? "").utf8.count + #expect( + byteCount < maxCanon, + Comment( + rawValue: "et command is \(byteCount) bytes for a \(session.count)-character " + + "session name, over the \(maxCanon)-byte canonical line limit" + ) + ) + } + } + + /// The session name reaches the remote shell as one word even when it contains a space or a + /// quote. Dropping the resolver moved this responsibility onto this profile's own quoting. + @Test func etQuotesTheSessionNameItTypes() { + let argv = RemoteTmuxETTransportProfile(port: 2039).controlStreamArgv( + host: RemoteTmuxHost(destination: "user@host"), + sessionName: "work 'session'; touch /tmp/cmux-et-injection", + createIfMissing: false + ) + let command = argv.first(where: { $0.hasPrefix("exec ") }) ?? "" + // Quoted as data, so the shell cannot run the trailing command. + #expect(command.contains("touch /tmp/cmux-et-injection")) + #expect(!command.contains("; touch /tmp/cmux-et-injection'\"")) + #expect(command.hasSuffix("'work '\\''session'\\''; touch /tmp/cmux-et-injection'")) + } + @Test func etCanTargetAServerWhoseTerminalIsNotOnThePath() { let profile = RemoteTmuxETTransportProfile( port: 2022, remoteTerminalPath: "/usr/local/bin/etterminal" diff --git a/docs/remote-tmux-transport-seam.md b/docs/remote-tmux-transport-seam.md index e53dc175c606..f3bf65eba9aa 100644 --- a/docs/remote-tmux-transport-seam.md +++ b/docs/remote-tmux-transport-seam.md @@ -54,17 +54,20 @@ protocol RemoteTmuxTransport: Sendable { implementation wraps a persistent-session transport. For ET that is: ``` -et --command 'exec -CC attach-session -t ' [--port N] @ +et --command 'exec tmux -CC attach-session -t ' [--port N] @ ``` Four details are load-bearing for any such transport: - **Run the command, do not open a shell.** ET's `--command` runs one command and exits. Prefix it with `exec` so the remote process tree does not keep a shell parent around. -- **The remote command still needs the tmux resolver.** A non-login remote shell often has - a minimal `PATH`, which is why `RemoteTmuxHost.tmuxRemoteCommand` wraps `tmux` in a - resolver script. That is a property of the remote shell, not of ssh, so it applies here - identically. +- **The command has to fit on one line, so et does not get the tmux resolver.** ET does not + exec the command: it types it into a login shell and appends `; exit`. That shell reads from + a pty in canonical mode, which delivers at most `MAX_CANON` (1024 on macOS) bytes per line, + and ssh's `PATH` resolver is about 1113 bytes. The line never completes, so the shell runs + nothing, the stream stays silent, and the attach dies on a timeout with nothing to explain + it. Plain `tmux` is both short enough and correct here, because a login shell already has + the user's full `PATH` — ssh is the opposite case, a non-login shell with a minimal one. - **The port is per-transport and per-host.** Keep it configurable instead of assuming ssh's 22. - **Never pass a flag that kills other sessions.** ET's `-x` / `--kill-other-sessions` diff --git a/scripts/remote-tmux-et-e2e.sh b/scripts/remote-tmux-et-e2e.sh index d787f39762b0..430b9f1a3929 100755 --- a/scripts/remote-tmux-et-e2e.sh +++ b/scripts/remote-tmux-et-e2e.sh @@ -55,29 +55,51 @@ log "=== attaching session '$SESSION' over the et transport" # mirroring whatever else happens to be on the host's default tmux server. # `cmux rpc` is the raw v2 call; a named attach avoids mirroring whatever else happens to # be on the host's default tmux server. +# +# The etserver port goes in transport_port, not port. `port` is the ssh port, and one-shot +# discovery still rides ssh, so putting 2039 there sends ssh at the etserver and it answers +# with `kex_exchange_identification: Connection reset by peer`. RESULT="$(cli rpc remote.tmux.attach \ - "{\"host\":\"$HOST\",\"session\":\"$SESSION\",\"port\":$PORT,\"transport\":\"et\"}" 2>&1)" + "{\"host\":\"$HOST\",\"session\":\"$SESSION\",\"transport\":\"et\",\"transport_port\":$PORT}" 2>&1)" log "attach result: $(printf '%s' "$RESULT" | head -c 300)" -# The decisive evidence is a live et process carrying tmux for this host, spawned by -# the app rather than by this script. -et_stream_live() { - pgrep -f "attach-session" 2>/dev/null | while read -r p; do - ps -o command= -p "$p" 2>/dev/null | grep -q "et " && echo x - done | grep -q x -} -if await "an et-carried control stream spawned by cmux" 60 et_stream_live; then - pass "cmux spawned a control stream over et" +case "$RESULT" in + *'"attached"'*) pass "the attach returned a result rather than an error" ;; + *) fail "attach did not succeed: $(printf '%s' "$RESULT" | head -c 120)" ;; +esac + +# The decisive evidence is the app's own view of the stream, not a process listing. A live +# process proves something was spawned; only these fields prove the control protocol crossed +# et and was understood — the two bugs this harness found both left a live process behind. +# +# enter the ESC P 1000 p handshake was parsed. cmux withholds commands until it arrives, +# and et delivers it mid-line behind the login shell's echo. +# windows a real `list-windows` result came back over the stream and was applied. +state() { cli rpc remote.tmux.state "{\"host\":\"$HOST\",\"session\":\"$SESSION\"}"; } +stream_entered() { state | grep -q '"enter_received" : true'; } +stream_has_windows() { state | grep -qE '"window_count" : [1-9]'; } + +if await "the control stream to reach control mode over et" 60 stream_entered; then + pass "cmux parsed the control-mode handshake over et" +else + fail "no control-mode handshake over et (enter_received stayed false)" +fi +if await "a window to arrive over the et-carried stream" 60 stream_has_windows; then + pass "tmux windows arrived over the et-carried stream" else - fail "no et-carried control stream appeared" + fail "no windows arrived over the et-carried stream" fi -# And it must be under a pty: a bare pipe spawn is exactly what produces no output. -pty_wrapped() { pgrep -fl "script -q /dev/null" >/dev/null 2>&1; } -if pty_wrapped; then - pass "the transport was spawned under a pseudo-terminal" +# And the transport must be this host's et, under a pty: a bare pipe spawn is exactly what +# produces no output. Match the whole shape so another agent's `script` or `et` cannot pass +# this for us. +pty_wrapped_et() { + pgrep -f "script -q /dev/null .*et -p $PORT .*$HOST" >/dev/null 2>&1 +} +if pty_wrapped_et; then + pass "this host's et was spawned under a pseudo-terminal" else - fail "no pty wrapper around the transport (et emits nothing on pipes)" + fail "no pty-wrapped et for $HOST:$PORT (et emits nothing on pipes)" fi log "=== $FAILURES failed check(s)" diff --git a/scripts/remote-tmux-et-host.sh b/scripts/remote-tmux-et-host.sh index a57062bd8152..40ae0f6ffb74 100755 --- a/scripts/remote-tmux-et-host.sh +++ b/scripts/remote-tmux-et-host.sh @@ -57,10 +57,33 @@ if ! nc -z 127.0.0.1 "$PORT" 2>/dev/null; then exit 1 fi +# A session for the mirror to attach to, on the server both transports reach. +# +# It has to be the server a plain login shell resolves to, because cmux splits the work: +# one-shot commands like the `has-session` check before an attach ride ssh, while the +# control stream rides et. Putting the session on a private TMUX_TMPDIR isolates it from +# the ssh side, so the attach fails with "can't find session" even though the session +# exists. A real et host has both transports landing on the same default server, so the +# harness matches that. +SESSION="${CMUX_ET_SESSION:-etmirror}" +OWNED="$DIR/owned-sessions" +if tmux has-session -t "$SESSION" 2>/dev/null; then + # Never adopt a session this harness did not create: teardown kills what it owns, and a + # developer's own session of the same name must survive. + grep -qxF "$SESSION" "$OWNED" 2>/dev/null \ + || { echo "tmux session '$SESSION' already exists and is not ours; set CMUX_ET_SESSION" >&2; exit 1; } +else + tmux new-session -d -s "$SESSION" -c "$HOME" 2>>"$DIR/logs/start.out" \ + || { echo "could not create tmux session $SESSION" >&2; exit 1; } + echo "$SESSION" >>"$OWNED" +fi +SOCKET="$(tmux display-message -p -t "$SESSION" '#{socket_path}' 2>/dev/null)" + cat <' $USER@127.0.0.1 INFO From 6c6cb68760ec8677cb841fabb84f26e5d4fd9b95 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 10:01:56 -0700 Subject: [PATCH 03/23] remote-tmux: put the transport in a host's identity and recover a stalled one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two faults from review, both real and both invisible to the end-to-end run, which exercises one host over one transport that works. `connectionHash` was built from destination, ssh port, and identity file, and it keys the attach single-flight, the transport registry, and matching a mirror to a host. So an ssh host and an et host at one destination were the same endpoint, as were two et hosts on different etserver ports, and an attach could be handed a cached connection whose profile or port was wrong. The transport and its port are in the fingerprint now, appended only for a non-default transport so a plain ssh host keeps the hash it has today — it names the shared master's socket path and persisted mirror state, and moving it would orphan both. `probeLiveness` had no caller outside a test. A transport that reconnects internally never delivers the EOF that drives ssh recovery: during a network change its process stays up and the stream pauses, which is the behavior worth having but leaves a wedged transport looking exactly like an idle one. Nothing else in the lifecycle can tell those apart, so a wedged et connection stayed `.connected` forever and the mirror froze with no error and no retry. A probe now runs while such a transport is connected, starting on `.enter` and cancelled with the rest of the scheduled work, and a stall routes through the existing `beginReconnecting()` rather than a second reconnect path — the remote session is likely still there, it is the client that is wedged, so this recovers rather than ends. ssh is excluded and tested to stay excluded: it gets its EOF, and probing an idle ssh stream would add traffic and a failure mode where there is none. `processGeneration` is readable across the type's extensions so a probe answered after a respawn can tell its answer describes a stream that no longer exists. Without that check a late answer would tear down the healthy connection that replaced it. The harness needed the same lesson the attach did: its `remote.tmux.state` lookup omitted the transport, so once identity included it the lookup matched nothing and reported the stream as never having reached control mode. Both e2e regressions today came from a call that under-specified the endpoint. --- ...RemoteTmuxControlConnection+Commands.swift | 53 +++++++++++++ Sources/RemoteTmuxControlConnection.swift | 34 +++++++- Sources/RemoteTmuxHost.swift | 15 +++- .../RemoteTmuxProxyTransportRetryTests.swift | 79 ++++++++++++++++++- scripts/remote-tmux-et-e2e.sh | 8 +- 5 files changed, 183 insertions(+), 6 deletions(-) diff --git a/Sources/RemoteTmuxControlConnection+Commands.swift b/Sources/RemoteTmuxControlConnection+Commands.swift index f9140b0ec681..b4009c76566c 100644 --- a/Sources/RemoteTmuxControlConnection+Commands.swift +++ b/Sources/RemoteTmuxControlConnection+Commands.swift @@ -110,6 +110,59 @@ extension RemoteTmuxControlConnection { return sendTracked("display-message -p cmux-liveness", completion: completion) } + /// Checks a stalled-but-alive control stream and recovers it. + /// + /// A transport that reconnects internally never delivers the EOF that drives ssh recovery: + /// during a network change its process stays up and the stream simply pauses. That is the + /// behavior worth having, but it means a transport that is alive and *not* recovering looks + /// exactly like one that is idle. Nothing else in the lifecycle can tell those apart, so + /// without this check a wedged et connection stays `.connected` forever and the mirror + /// freezes with no error and no retry. + /// + /// Only reachable for `reconnectsInternally` transports: ssh gets its EOF and must keep its + /// existing behavior exactly, including staying quiet on an idle stream. + /// + /// - Parameter completion: `true` if the stream answered (or the check did not apply), and + /// `false` if it was found wedged and recovery was started. + func checkLivenessAndRecoverIfStalled(completion: ((Bool) -> Void)? = nil) { + guard transportProfile.reconnectsInternally, connectionState == .connected, !exited else { + completion?(true) + return + } + let generation = processGeneration + // A probe that cannot even be enqueued means the stream is already unusable. + let enqueued = probeLiveness { [weak self] answered in + guard let self else { return } + guard generation == self.processGeneration else { + // A respawn overtook this probe; its answer says nothing about the live stream. + completion?(true) + return + } + if answered { + completion?(true) + } else { + self.recoverFromStalledTransport() + completion?(false) + } + } + if !enqueued { + recoverFromStalledTransport() + completion?(false) + } + } + + /// Replaces a transport that is alive but no longer carrying the protocol. + /// + /// Recovery is a respawn rather than an end: the remote session is very likely still there — + /// it is the client that is wedged — so the mirror should be reconnected, not torn down. This + /// routes through the same `beginReconnecting()` path as an ssh transport loss so there is one + /// reconnect implementation rather than a second one for this case. + private func recoverFromStalledTransport() { + guard connectionState == .connected else { return } + record("liveness-stalled") + beginReconnecting() + } + func failPendingTrackedSends() { let completions = Array(trackedSendCompletions.values) trackedSendCompletions.removeAll() diff --git a/Sources/RemoteTmuxControlConnection.swift b/Sources/RemoteTmuxControlConnection.swift index 3ed7ffdac00b..fb48dd2bfadb 100644 --- a/Sources/RemoteTmuxControlConnection.swift +++ b/Sources/RemoteTmuxControlConnection.swift @@ -123,7 +123,10 @@ final class RemoteTmuxControlConnection { private var stderrTask: Task? private var parser = RemoteTmuxControlStreamParser() private var ingestTask: Task? - private var processGeneration: UInt64 = 0 + /// Bumped on every spawn. Readable across the type's extensions so a completion that + /// outlived its process — a liveness probe answered after a respawn, say — can tell that its + /// answer describes a stream that no longer exists. Writable only here. + private(set) var processGeneration: UInt64 = 0 var pendingCommands: [CommandKind] = [] var windowListRequestInFlight = false var windowListRequestDirty = false @@ -155,6 +158,13 @@ final class RemoteTmuxControlConnection { /// attempts); cancelled on `stop()` / genuine end so a dead connection stops /// retrying. private var reconnectTask: Task? + /// Periodic liveness probe for transports that reconnect internally (see + /// ``checkLivenessAndRecoverIfStalled(completion:)``). Nil for ssh, which gets an EOF instead. + private var livenessTask: Task? + /// How often to ask a self-reconnecting transport whether it is still carrying the protocol. + /// Long enough that an ordinary reconnect finishes untouched, short enough that a wedged + /// mirror is not left silently frozen. + static var livenessProbeIntervalSeconds: UInt64 = 30 /// Number of reconnect attempts since the last successful connect, driving the /// capped exponential backoff. Reset to 0 on a successful connect. private var reconnectAttemptCount = 0 @@ -567,6 +577,8 @@ final class RemoteTmuxControlConnection { failPendingCommandTransactions() reconnectTask?.cancel() reconnectTask = nil + livenessTask?.cancel() + livenessTask = nil resetWindowListRequestCoalescing() cancelSizingFollowUps() pendingPostAttachAction = nil @@ -766,6 +778,25 @@ final class RemoteTmuxControlConnection { // MARK: - Reconnect + /// Starts the stall monitor for a transport that owns its own reconnection. + /// + /// ssh is deliberately excluded: its stream ends on transport loss, `handleStreamEnd` already + /// recovers from that, and probing an idle ssh stream would add traffic and a failure mode + /// where today there is none. + private func startLivenessMonitorIfNeeded() { + guard transportProfile.reconnectsInternally else { return } + livenessTask?.cancel() + let interval = Self.livenessProbeIntervalSeconds + livenessTask = Task { [weak self] in + while !Task.isCancelled { + try? await Task.sleep(nanoseconds: interval * 1_000_000_000) + if Task.isCancelled { return } + guard let self else { return } + await MainActor.run { self.checkLivenessAndRecoverIfStalled() } + } + } + } + /// Freezes the mirror and reconnects after an unusable control stream. func beginReconnecting() { guard connectionState == .connected || connectionState == .connecting else { return } @@ -844,6 +875,7 @@ final class RemoteTmuxControlConnection { if connectionState != .connected { let wasReconnecting = connectionState == .reconnecting connectionState = .connected + startLivenessMonitorIfNeeded() // Only a first attach needs the rows-minus-one redraw kick. A // reconnect keeps the existing tmux grid and replaces the mirror // with an authoritative full-history seed; kicking after that seed diff --git a/Sources/RemoteTmuxHost.swift b/Sources/RemoteTmuxHost.swift index 7f9e4eb13c97..1f82e864ff46 100644 --- a/Sources/RemoteTmuxHost.swift +++ b/Sources/RemoteTmuxHost.swift @@ -106,7 +106,20 @@ struct RemoteTmuxHost: Sendable, Equatable, Identifiable { /// distinct endpoints must never collapse onto one socket and risk routing a /// command to the wrong server. var connectionHash: String { - let fingerprint = "\(destination)\u{1f}\(port.map(String.init) ?? "")\u{1f}\(identityFile ?? "")" + // The transport and its port belong in the fingerprint because they decide what the + // control stream actually is. Everything keyed by this hash — the attach single-flight, + // the transport registry, matching a mirror to a host — would otherwise treat an ssh + // host and an et host at the same destination as one endpoint, and hand an attach a + // cached connection whose profile or port is wrong. Two et hosts on different + // etserver ports collide the same way. + // + // Only appended for a non-default transport, so a plain ssh host keeps the hash it has + // today: it names the shared master's socket path and persisted mirror state, and + // changing it for existing hosts would orphan both. + var fingerprint = "\(destination)\u{1f}\(port.map(String.init) ?? "")\u{1f}\(identityFile ?? "")" + if transport != .ssh || transportPort != nil { + fingerprint += "\u{1f}\(transport.rawValue)\u{1f}\(transportPort.map(String.init) ?? "")" + } var hash: UInt64 = 0xcbf2_9ce4_8422_2325 // FNV offset basis for byte in fingerprint.utf8 { hash ^= UInt64(byte) diff --git a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift index 915bb90dabea..c99794117ea6 100644 --- a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift +++ b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift @@ -112,9 +112,10 @@ import Testing createIfMissing: Bool ) -> [String] { // `--command` runs one command and exits, and `exec` keeps a shell parent out of - // the remote process tree. The tmux resolver is still required: a non-login - // remote shell has a minimal PATH, which is a property of the remote shell and - // not of ssh, so it applies to every transport identically. + // the remote process tree. This stand-in keeps the resolver because it says nothing + // about how its command reaches the remote shell; the real et profile drops it, + // since et types the command into a login shell that both resolves PATH itself and + // cannot read a line that long. let remote = RemoteTmuxHost.tmuxRemoteCommand( arguments: ["-CC", createIfMissing ? "new-session" : "attach-session", "-t", sessionName] ) @@ -379,6 +380,78 @@ import Testing #expect(command.hasSuffix("'work '\\''session'\\''; touch /tmp/cmux-et-injection'")) } + /// The hash keys the attach single-flight, the transport registry, and mirror-to-host + /// matching, so anything that changes what the control stream *is* has to be in it. Without + /// the transport an ssh host and an et host at one destination are the same endpoint, and an + /// attach can be handed a cached connection running the wrong profile or port. + @Test func theConnectionHashSeparatesTransportsAndTheirPorts() { + let ssh = RemoteTmuxHost(destination: "user@host") + let et = RemoteTmuxHost(destination: "user@host", transport: .et) + let et2039 = RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2039) + let et2040 = RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2040) + + #expect(ssh.connectionHash != et.connectionHash) + #expect(et.connectionHash != et2039.connectionHash) + #expect(et2039.connectionHash != et2040.connectionHash, "etserver port must separate endpoints") + // The ssh port is a different axis from the transport port and must not alias it. + #expect( + RemoteTmuxHost(destination: "user@host", port: 2039).connectionHash != et2039.connectionHash + ) + } + + /// And a plain ssh host keeps the hash it has today. It names the shared master's socket path + /// and persisted mirror state, so moving it would orphan both on upgrade. + @Test func theConnectionHashIsUnchangedForAnSSHHost() { + // Naming ssh explicitly, with no transport port, is the same endpoint as saying nothing — + // which is what keeps an existing host's socket path and persisted state addressable. + #expect( + RemoteTmuxHost(destination: "user@host").connectionHash + == RemoteTmuxHost(destination: "user@host", transport: .ssh).connectionHash + ) + #expect( + RemoteTmuxHost(destination: "user@host", port: 22, identityFile: "/k").connectionHash + == RemoteTmuxHost( + destination: "user@host", port: 22, identityFile: "/k", transport: .ssh + ).connectionHash + ) + } + + /// A self-reconnecting transport that is alive but no longer answering has to be recovered, + /// not waited on. There is no EOF coming — that is the whole point of such a transport — so + /// before this the connection stayed `.connected` and the mirror froze permanently. + /// + /// `.enter` is delivered through the real message path rather than a test-only setter, so the + /// transition under test is the one production takes. + @MainActor @Test func aStalledSelfReconnectingTransportIsRecoveredRatherThanLeftConnected() { + let connection = RemoteTmuxControlConnection( + host: RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2039), + sessionName: "work" + ) + connection.handle(.enter) + #expect(!connection.snapshot().recentEvents.contains("liveness-stalled")) + + // Nothing is attached to carry the probe, which is what a wedged transport looks like + // from here: alive as far as anyone can see, unable to answer. + var reported: Bool? + connection.checkLivenessAndRecoverIfStalled { reported = $0 } + #expect(reported == false, "a stream that cannot answer must be reported as stalled") + #expect(connection.snapshot().recentEvents.contains("liveness-stalled")) + } + + /// ssh must be untouched by all of this: it gets an EOF, `handleStreamEnd` already recovers, + /// and probing an idle ssh stream would add traffic and a new way to fail. + @MainActor @Test func anSSHTransportIsNeverProbedForStalls() { + let connection = RemoteTmuxControlConnection( + host: RemoteTmuxHost(destination: "user@host"), + sessionName: "work" + ) + connection.handle(.enter) + var reported: Bool? + connection.checkLivenessAndRecoverIfStalled { reported = $0 } + #expect(reported == true, "ssh is out of scope for the stall check") + #expect(!connection.snapshot().recentEvents.contains("liveness-stalled")) + } + @Test func etCanTargetAServerWhoseTerminalIsNotOnThePath() { let profile = RemoteTmuxETTransportProfile( port: 2022, remoteTerminalPath: "/usr/local/bin/etterminal" diff --git a/scripts/remote-tmux-et-e2e.sh b/scripts/remote-tmux-et-e2e.sh index 430b9f1a3929..efefb7818224 100755 --- a/scripts/remote-tmux-et-e2e.sh +++ b/scripts/remote-tmux-et-e2e.sh @@ -75,7 +75,13 @@ esac # enter the ESC P 1000 p handshake was parsed. cmux withholds commands until it arrives, # and et delivers it mid-line behind the login shell's echo. # windows a real `list-windows` result came back over the stream and was applied. -state() { cli rpc remote.tmux.state "{\"host\":\"$HOST\",\"session\":\"$SESSION\"}"; } +# The transport and its port are part of the endpoint's identity, so a lookup that omits them +# addresses a different endpoint and quietly matches nothing — which reads as "the stream never +# reached control mode" rather than "you asked about the wrong connection". +state() { + cli rpc remote.tmux.state \ + "{\"host\":\"$HOST\",\"session\":\"$SESSION\",\"transport\":\"et\",\"transport_port\":$PORT}" +} stream_entered() { state | grep -q '"enter_received" : true'; } stream_has_windows() { state | grep -qE '"window_count" : [1-9]'; } From 0b5ee22abc175cc8de0746e5690ea6d5cb265c59 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 11:10:44 -0700 Subject: [PATCH 04/23] remote-tmux: detect an unanswered liveness probe, and guard against new polling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The liveness monitor could not detect the stall it exists for. ET can accept stdin while producing no control output, and the probe had no response deadline, so probes were written, never answered, accumulated, and the connection sat `.connected` forever. The deadline is now the next probe's due time rather than a second clock: probe N must be answered before N+1 is due, which is a generous bound on a local round-trip. A respawn clears the flag so a stale outstanding probe cannot condemn a fresh stream. `connectionHash` gains the case its tests were missing — `.ssh` with a transport port, which the discriminator admits and nothing else covered. It must not alias the plain ssh host, the ssh host whose *ssh* port matches, or the et host on the same transport port. Also adds a guard against the class of bug this work kept producing. A reviewer found a `Task.sleep` backoff in the login waiter whose own comment claimed "there is no event to subscribe to"; the event was the ControlMaster socket being created and `FileWatcher` had been in the tree the whole time. Nothing failed when that shipped, so nothing would fail next time. `scripts/lint-remote-tmux-no-polling.sh` fails on a new time-based wait in remote-tmux sources and names the edges this codebase actually has. It blesses nothing: six pre-existing waits are recorded in a baseline, matched by file and symbol so unrelated edits do not trip it, and the sizing debounces there are worth revisiting since sizing is supposed to converge on tmux's ordered acknowledgements rather than a clock. Two exceptions carry the reason no edge exists. --- ...RemoteTmuxControlConnection+Commands.swift | 15 +++ Sources/RemoteTmuxControlConnection.swift | 4 + .../RemoteTmuxProxyTransportRetryTests.swift | 38 ++++++++ scripts/lint-remote-tmux-no-polling.sh | 94 +++++++++++++++++++ scripts/remote-tmux-polling-baseline.txt | 6 ++ 5 files changed, 157 insertions(+) create mode 100755 scripts/lint-remote-tmux-no-polling.sh create mode 100644 scripts/remote-tmux-polling-baseline.txt diff --git a/Sources/RemoteTmuxControlConnection+Commands.swift b/Sources/RemoteTmuxControlConnection+Commands.swift index b4009c76566c..e73d0bd92503 100644 --- a/Sources/RemoteTmuxControlConnection+Commands.swift +++ b/Sources/RemoteTmuxControlConnection+Commands.swift @@ -130,8 +130,22 @@ extension RemoteTmuxControlConnection { return } let generation = processGeneration + // An unanswered previous probe IS the stall. ET can accept stdin while producing no + // control output, so a probe can be written and simply never answered — without this the + // probes accumulate, every one of them still pending, and the connection sits + // `.connected` forever. The deadline is the next tick rather than a second timer: probe N + // must be answered before probe N+1 is due, which is a generous bound on a local + // round-trip and needs no clock of its own. + if livenessProbeOutstanding { + record("liveness-unanswered") + recoverFromStalledTransport() + completion?(false) + return + } + livenessProbeOutstanding = true // A probe that cannot even be enqueued means the stream is already unusable. let enqueued = probeLiveness { [weak self] answered in + self?.livenessProbeOutstanding = false guard let self else { return } guard generation == self.processGeneration else { // A respawn overtook this probe; its answer says nothing about the live stream. @@ -146,6 +160,7 @@ extension RemoteTmuxControlConnection { } } if !enqueued { + livenessProbeOutstanding = false recoverFromStalledTransport() completion?(false) } diff --git a/Sources/RemoteTmuxControlConnection.swift b/Sources/RemoteTmuxControlConnection.swift index fb48dd2bfadb..85ffd2f5a291 100644 --- a/Sources/RemoteTmuxControlConnection.swift +++ b/Sources/RemoteTmuxControlConnection.swift @@ -161,6 +161,9 @@ final class RemoteTmuxControlConnection { /// Periodic liveness probe for transports that reconnect internally (see /// ``checkLivenessAndRecoverIfStalled(completion:)``). Nil for ssh, which gets an EOF instead. private var livenessTask: Task? + /// Whether a liveness probe is still waiting for its answer. The next probe's due time is the + /// previous one's deadline, so this is what turns "no answer" into a detected stall. + var livenessProbeOutstanding = false /// How often to ask a self-reconnecting transport whether it is still carrying the protocol. /// Long enough that an ordinary reconnect finishes untouched, short enough that a wedged /// mirror is not left silently frozen. @@ -579,6 +582,7 @@ final class RemoteTmuxControlConnection { reconnectTask = nil livenessTask?.cancel() livenessTask = nil + livenessProbeOutstanding = false resetWindowListRequestCoalescing() cancelSizingFollowUps() pendingPostAttachAction = nil diff --git a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift index c99794117ea6..9acb5e5c0d25 100644 --- a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift +++ b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift @@ -399,6 +399,24 @@ import Testing ) } + /// `.ssh` with a transport port is the case the discriminator's condition admits but nothing + /// else does: for ssh the endpoint's port is `port`, so a transport port is meaningless there. + /// It must still be its own endpoint rather than aliasing the plain ssh host or the ssh host + /// whose *ssh* port happens to match. + @Test func anSSHHostWithATransportPortIsItsOwnEndpoint() { + let plain = RemoteTmuxHost(destination: "user@host") + let sshWithTransportPort = RemoteTmuxHost(destination: "user@host", transport: .ssh, transportPort: 2039) + let sshOnThatPort = RemoteTmuxHost(destination: "user@host", port: 2039) + + #expect(sshWithTransportPort.connectionHash != plain.connectionHash) + #expect(sshWithTransportPort.connectionHash != sshOnThatPort.connectionHash) + #expect( + sshWithTransportPort.connectionHash + != RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2039).connectionHash, + "the transport itself must still separate these" + ) + } + /// And a plain ssh host keeps the hash it has today. It names the shared master's socket path /// and persisted mirror state, so moving it would orphan both on upgrade. @Test func theConnectionHashIsUnchangedForAnSSHHost() { @@ -438,6 +456,26 @@ import Testing #expect(connection.snapshot().recentEvents.contains("liveness-stalled")) } + /// A probe that is written but never answered is the stall this monitor exists for: ET can + /// accept stdin while producing no control output. Before the deadline, probes accumulated and + /// the connection stayed `.connected` forever — the monitor could not detect the very case it + /// was added for. + @MainActor @Test func anUnansweredProbeIsTreatedAsAStall() { + let connection = RemoteTmuxControlConnection( + host: RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2039), + sessionName: "work" + ) + connection.handle(.enter) + // Stand in for a probe that was written and never came back. + connection.livenessProbeOutstanding = true + + var reported: Bool? + connection.checkLivenessAndRecoverIfStalled { reported = $0 } + #expect(reported == false) + #expect(connection.snapshot().recentEvents.contains("liveness-unanswered")) + #expect(connection.snapshot().recentEvents.contains("liveness-stalled")) + } + /// ssh must be untouched by all of this: it gets an EOF, `handleStreamEnd` already recovers, /// and probing an idle ssh stream would add traffic and a new way to fail. @MainActor @Test func anSSHTransportIsNeverProbedForStalls() { diff --git a/scripts/lint-remote-tmux-no-polling.sh b/scripts/lint-remote-tmux-no-polling.sh new file mode 100755 index 000000000000..7d1361fb3915 --- /dev/null +++ b/scripts/lint-remote-tmux-no-polling.sh @@ -0,0 +1,94 @@ +#!/bin/bash +# ============================================================================ +# Fails if remote-tmux gains a new sleep, timer, or poll loop. +# +# Remote-tmux waits on things constantly — a control-mode reply, a shared master, a person +# finishing a login — and every one of those has an event to wait on. A timer instead of the +# event is not a style preference: its interval is dead time a frozen mirror spends after the +# thing it was waiting for already happened, and it can miss the event entirely. +# +# This exists because knowing that is not enough. A reviewer caught a `Task.sleep` backoff in +# the login waiter that had shipped with a comment claiming "there is no event to subscribe +# to" — the event was the ControlMaster socket being created, and `FileWatcher` had been in +# the tree the whole time. Nothing failed when that went in, so nothing will fail the next +# time either, unless something checks. +# +# Adding a wait that genuinely has no edge is allowed, but it has to be listed below with the +# reason, which makes the exception visible in review instead of implicit in a diff. +# +# Usage: scripts/lint-remote-tmux-no-polling.sh +# Exit 0 when clean, 1 with the offending file:line otherwise. +# ============================================================================ +set -uo pipefail + +cd "$(dirname "$0")/.." || exit 1 + +# Product sources only. Tests may need to drive time directly, and scripts are harnesses +# where polling an external process is often the only option available. +SCOPE=(Sources/RemoteTmux*.swift Sources/RemoteTmuxController+*.swift) + +# Primitives that make a wait time-based rather than event-based. +PATTERN='Task\.sleep|Thread\.sleep|usleep\(|DispatchQueue\.[A-Za-z.]*asyncAfter|DispatchSourceTimer|Timer\.scheduledTimer|ContinuousClock\(\)\.sleep' + +# Waits that predate this guard, recorded so it blocks NEW ones without pretending the +# existing ones are all fine. Several are worth revisiting — the sizing debounces in +# particular, since sizing convergence is supposed to be driven by tmux's ordered +# %begin/%end acknowledgements rather than a wall clock. Removing an entry from this list +# is progress; adding one needs the reason to say why no edge exists. +BASELINE_FILE="scripts/remote-tmux-polling-baseline.txt" + +# Exceptions introduced deliberately, with the reason no edge exists. +ALLOW=( + "Sources/RemoteTmuxControlConnection.swift:startLivenessMonitorIfNeeded|A transport that reconnects internally emits NO EOF for a network drop: its process stays up and the stream pauses, so a wedged transport is indistinguishable from an idle one. There is no event for 'still carrying the protocol' — the only way to know is to ask, so this probes rather than waits." + "Sources/RemoteTmuxControlConnection.swift:scheduleReconnectAttempt|Reconnect backoff for a host that is unreachable. The edge would be 'the host came back', which nothing local can observe; retrying IS the observation." +) + +fail=0 +while IFS= read -r hit; do + [ -z "$hit" ] && continue + file="${hit%%:*}" + rest="${hit#*:}" + line="${rest%%:*}" + + # Find the enclosing func by walking back to the nearest declaration. + symbol="$(awk -v n="$line" 'NR<=n && /func [A-Za-z_]/ { s=$0 } END { print s }' "$file" \ + | sed -E 's/.*func ([A-Za-z_][A-Za-z0-9_]*).*/\1/')" + + allowed=0 + for entry in "${ALLOW[@]}"; do + key="${entry%%|*}" + if [ "$key" = "$file:$symbol" ]; then allowed=1; break; fi + done + # Pre-existing waits are matched by file+symbol, not line number, so unrelated edits above + # them do not turn into lint failures. + if [ "$allowed" -eq 0 ] && [ -f "$BASELINE_FILE" ] \ + && grep -qxF "$file:$symbol" "$BASELINE_FILE"; then + allowed=1 + fi + + if [ "$allowed" -eq 0 ]; then + echo "lint-remote-tmux-no-polling: $file:$line — time-based wait in '$symbol'" >&2 + echo " $(sed -n "${line}p" "$file" | sed 's/^[[:space:]]*//')" >&2 + fail=1 + fi +done < <(grep -nE "$PATTERN" "${SCOPE[@]}" 2>/dev/null \ + | awk -F: '{ body = substr($0, index($0, $3)); sub(/^[[:space:]]+/, "", body); if (body !~ /^\/\//) print }' || true) + +if [ "$fail" -ne 0 ]; then + cat >&2 <<'EOF' + +Waiting on a timer means the event that ends the wait was not used. Find the edge first: + - a control-mode reply -> sendTracked / the %begin/%end correlation + - a file or socket appearing -> FileWatcher (watches the parent directory too, so + creation is visible for a path that does not exist yet) + - terminal output, e.g. a marker -> the per-surface PTY tee detectors + - a workspace closing -> the TabManager close path calls the controller +An event-driven wait must also check its condition once up front: an edge that already +happened is never delivered. + +If there is genuinely no edge, add the symbol to ALLOW in this script with the reason. +EOF + exit 1 +fi + +echo "lint-remote-tmux-no-polling: ok (${#ALLOW[@]} documented, $(wc -l < "$BASELINE_FILE" 2>/dev/null | tr -d ' ') baselined)" diff --git a/scripts/remote-tmux-polling-baseline.txt b/scripts/remote-tmux-polling-baseline.txt new file mode 100644 index 000000000000..d98fbac288fc --- /dev/null +++ b/scripts/remote-tmux-polling-baseline.txt @@ -0,0 +1,6 @@ +Sources/RemoteTmuxControlConnection+Sizing.swift:sendPerWindowRedrawKick +Sources/RemoteTmuxControlConnection+Sizing.swift:sendSessionRedrawKick +Sources/RemoteTmuxControlConnection+Sizing.swift:setClientSize +Sources/RemoteTmuxControlConnection+Sizing.swift:setWindowSize +Sources/RemoteTmuxSSHTransport.swift:killSessions +Sources/RemoteTmuxWindowMirror.swift:establishPaneKeyFocusWhenMounted From 4e63a2a75f27873b55f2aa8cd1037ceb2e63d608 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 11:31:37 -0700 Subject: [PATCH 05/23] remote-tmux: stop inferring session death from end-of-stream MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured against et 6.2.11+7: restarting only `etserver` ends the control stream while `tmux has-session` still succeeds. cmux read that as the session being over and removed the mirror, throwing away a session that was alive and reattachable. The rule it came from sounded right — a transport that reconnects internally does not end for a network drop, so its exit must mean the session ended — but EOF cannot distinguish "the transport died" from "the session died" for any transport. So it no longer tries: end-of-stream reconnects, and the reattach reports whether the session is gone, which is an answer rather than an inference. `forStreamEnd` takes no argument now; a parameter that no longer decides anything would only invite the same inference back. The model-based fuzz test caught this honestly. Its invariants encoded the old premise — an exit ends the session, and a stall never respawns — and both are now false: a stall respawns because a wedged transport will not recover itself. Rewritten to the properties that survive the change: a stall and an exit never *end* a connection, only a session found gone does, and a connection is never left both ended and alive. --- Sources/RemoteTmuxControlConnection.swift | 4 +- Sources/RemoteTmuxTransportRegistry.swift | 27 ++++--- .../RemoteTmuxProxyTransportRetryTests.swift | 70 +++++++++---------- 3 files changed, 51 insertions(+), 50 deletions(-) diff --git a/Sources/RemoteTmuxControlConnection.swift b/Sources/RemoteTmuxControlConnection.swift index 85ffd2f5a291..dab8291b129b 100644 --- a/Sources/RemoteTmuxControlConnection.swift +++ b/Sources/RemoteTmuxControlConnection.swift @@ -739,9 +739,7 @@ final class RemoteTmuxControlConnection { // reconnection: for ssh it is a transport loss cmux recovers from, but a // transport that reconnects internally does not end for a network drop, so its // exit is the session genuinely ending. - switch RemoteTmuxStreamEndDisposition.forStreamEnd( - reconnectsInternally: transportProfile.reconnectsInternally - ) { + switch RemoteTmuxStreamEndDisposition.forStreamEnd() { case .reconnect: // Keep the mirror frozen and reconnect. beginReconnecting() diff --git a/Sources/RemoteTmuxTransportRegistry.swift b/Sources/RemoteTmuxTransportRegistry.swift index aa7281c1eba9..d080a2d55ca6 100644 --- a/Sources/RemoteTmuxTransportRegistry.swift +++ b/Sources/RemoteTmuxTransportRegistry.swift @@ -84,22 +84,29 @@ protocol RemoteTmuxTransportProfile: Sendable { /// What end-of-stream on the control connection means. /// -/// cmux's recovery is built on stdout EOF: the stream ends, so respawn with backoff. That -/// is right for ssh, where a dropped connection ends the process. It is wrong for a -/// transport that owns its own reconnection: such a transport does not end for a network -/// drop — the stream pauses and resumes — so if it *does* end, it has genuinely exited and -/// the session is over. Respawning then would be cmux fighting the transport for ownership -/// of recovery, and the failure it must watch for instead is "alive but wedged". +/// cmux's recovery is built on stdout EOF: the stream ends, so respawn with backoff. +/// +/// The tempting rule is that a transport owning its own reconnection does not end for a network +/// drop, so its exit must mean the session ended. Measured against et 6.2.11+7, that is false: +/// restarting only `etserver` closes the stream while `tmux has-session` still succeeds. Acting on +/// it discarded mirrors whose sessions were alive and reattachable. +/// +/// EOF cannot distinguish "the transport died" from "the session died", for any transport, so this +/// does not try. Reattaching answers the question: the reconnect path already classifies a genuinely +/// gone session from what the reattach reports. The failure such a transport still needs watching +/// for is the one EOF never reports at all — alive but wedged. enum RemoteTmuxStreamEndDisposition: Sendable, Equatable { /// cmux owns recovery: respawn the transport with backoff. case reconnect /// The transport owned recovery, so its exit is terminal. case sessionOver - /// Decides from who owns reconnection. - static func forStreamEnd(reconnectsInternally: Bool) -> RemoteTmuxStreamEndDisposition { - reconnectsInternally ? .sessionOver : .reconnect - } + /// What EOF on the control stream means, for every transport: reconnect and find out. + /// + /// Deliberately takes no argument. It used to branch on who owns reconnection, and that branch + /// was wrong (see this type's documentation) — a parameter that no longer decides anything + /// would just invite the same inference back. + static func forStreamEnd() -> RemoteTmuxStreamEndDisposition { .reconnect } } /// A command to run before opening a connection to a host. diff --git a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift index 9acb5e5c0d25..91647976211d 100644 --- a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift +++ b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift @@ -186,20 +186,15 @@ import Testing // MARK: - Seam 2: who owns reconnection decides what EOF means - /// ssh ends when the connection drops, so EOF is cmux's cue to respawn. - @Test func endOfStreamOverSSHMeansReconnect() { - #expect( - RemoteTmuxStreamEndDisposition.forStreamEnd(reconnectsInternally: false) == .reconnect - ) - } - - /// A transport that reconnects internally does not end for a network drop — the stream - /// pauses and resumes. So if it ends, the session is genuinely over, and respawning - /// would be cmux fighting the transport for ownership of recovery. - @Test func endOfStreamOverAPersistentTransportMeansTheSessionIsOver() { - #expect( - RemoteTmuxStreamEndDisposition.forStreamEnd(reconnectsInternally: true) == .sessionOver - ) + /// EOF means reconnect, whoever owns reconnection, because EOF cannot say which of the + /// transport and the session died. + /// + /// This test previously asserted the opposite for a self-reconnecting transport, on the + /// reasoning that such a transport does not end for a network drop. Measured against + /// et 6.2.11+7: restarting only `etserver` ends the stream while `tmux has-session` still + /// succeeds, so that reasoning discarded live, reattachable sessions. + @Test func endOfStreamAlwaysMeansReconnectAndLetTheReattachDecide() { + #expect(RemoteTmuxStreamEndDisposition.forStreamEnd() == .reconnect) } // MARK: - Seam 3: the pre-connect hook @@ -505,8 +500,6 @@ import Testing let profile = RemoteTmuxETTransportProfile() #expect(profile.reconnectsInternally) #expect(profile.requiresPseudoTerminal) - #expect(RemoteTmuxStreamEndDisposition.forStreamEnd( - reconnectsInternally: profile.reconnectsInternally) == .sessionOver) } // MARK: - Seam 2: telling a stall from a death @@ -529,16 +522,16 @@ import Testing } /// The property that makes the probe necessary in the first place. + /// + /// A transport that reconnects internally produces no EOF for a network drop, so the failure it + /// can suffer — alive but no longer carrying the protocol — is one EOF never reports. That is + /// what the probe is for. EOF itself is not the discriminator it once looked like: it now means + /// reconnect for every transport, because it cannot say whether the transport or the session + /// ended (see endOfStreamAlwaysMeansReconnectAndLetTheReattachDecide). @Test func aStallIsNotADeathForATransportThatReconnectsItself() { let et = RemoteTmuxETTransportProfile() #expect(et.reconnectsInternally) - // EOF from such a transport means the session is genuinely over... - #expect(RemoteTmuxStreamEndDisposition.forStreamEnd( - reconnectsInternally: true) == .sessionOver) - // ...whereas ssh's EOF is cmux's cue to respawn. Same event, opposite meaning, which - // is why a stall has to be diagnosed by asking rather than by waiting for EOF. - #expect(RemoteTmuxStreamEndDisposition.forStreamEnd( - reconnectsInternally: false) == .reconnect) + #expect(!RemoteTmuxSSHTransportProfile().reconnectsInternally) } // MARK: - Runtime selection @@ -680,7 +673,6 @@ private struct SplitMix64 { #expect(profile.reconnectsInternally, "this suite is about internally-reconnecting transports") var model = Model() - let spawnsAtStart = model.spawnCount for step in 0..<40 { let event = Event.allCases[rng.int(0.. Date: Tue, 21 Jul 2026 11:34:10 -0700 Subject: [PATCH 06/23] remote-tmux: check what cmux believes about et, against et MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ET transport shipped with six defects sharing one cause: its load-bearing claims about EternalTerminal lived in a doc section and in comments, and nothing executed them. Unit tests cannot catch that class — they assert what cmux builds, which is the model, not what the other program does, and 77 of them passed while the transport could not carry a stream at all. So each claim is now a check against the real binary: the remote command runs in a login shell (which is why the ET argv drops ssh PATH resolver), a command longer than one canonical line is not delivered (which is why it has to), the control-mode enter DCS arrives mid-line behind the shell echo, and end-of-stream does not mean the session is gone. All four pass against 6.2.11+7. Parameterised by binary because ET is not as widely deployed as tmux and its behaviour moves between series — 7.x rewrote the pty input path that 6.x deadlocks on. The design doc now points at the script instead of asserting the properties itself. --- docs/remote-tmux-transport-seam.md | 7 +- scripts/remote-tmux-et-conformance.sh | 132 ++++++++++++++++++++++++++ 2 files changed, 138 insertions(+), 1 deletion(-) create mode 100755 scripts/remote-tmux-et-conformance.sh diff --git a/docs/remote-tmux-transport-seam.md b/docs/remote-tmux-transport-seam.md index f3bf65eba9aa..1a5e821b006a 100644 --- a/docs/remote-tmux-transport-seam.md +++ b/docs/remote-tmux-transport-seam.md @@ -57,7 +57,12 @@ implementation wraps a persistent-session transport. For ET that is: et --command 'exec tmux -CC attach-session -t ' [--port N] @ ``` -Four details are load-bearing for any such transport: +Four details are load-bearing for any such transport. Each one is a claim about a program cmux +does not control, so each one is a check in `scripts/remote-tmux-et-conformance.sh` rather than +only a paragraph here — every defect this transport shipped with was a claim in this list that +nothing executed. Run that script against every ET version you intend to support: 7.x rewrote the +pty input path that 6.x deadlocks on, so "et behaves like this" is version-scoped. + - **Run the command, do not open a shell.** ET's `--command` runs one command and exits. Prefix it with `exec` so the remote process tree does not keep a shell parent around. diff --git a/scripts/remote-tmux-et-conformance.sh b/scripts/remote-tmux-et-conformance.sh new file mode 100755 index 000000000000..6e48d8ad71d6 --- /dev/null +++ b/scripts/remote-tmux-et-conformance.sh @@ -0,0 +1,132 @@ +#!/bin/bash +# ============================================================================ +# Checks the things cmux believes about `et`, against `et`. +# +# The ET transport shipped with six defects that shared one cause: its load-bearing claims about +# EternalTerminal lived in a design doc and in comments, and nothing executed them. Unit tests +# could not catch any of them — they assert what cmux builds, which is the model, not what the +# other program does. 77 of them passed while the transport could not carry a stream at all. +# +# So each claim gets one check here. A claim that cannot be written as a check is an assumption, +# and belongs in the PR as one. +# +# Version matters more than usual. ET is not as widely deployed as tmux, and its behaviour does +# move between series: 7.x rewrote the pty input path that 6.x deadlocks on. Run this against +# every version you intend to support and treat a divergence as a finding. +# +# Usage: +# scripts/remote-tmux-et-conformance.sh # the et on PATH +# ET_CLIENT=/path/to/et ET_SERVER=/path/to/etserver \ +# ET_TERMINAL=/path/to/etterminal scripts/remote-tmux-et-conformance.sh +# +# Exit code is the number of failed checks (0 = every belief holds). +# ============================================================================ +set -uo pipefail + +cd "$(dirname "$0")/.." || exit 1 + +ET_CLIENT="${ET_CLIENT:-$(command -v et || true)}" +ET_SERVER="${ET_SERVER:-$(command -v etserver || true)}" +# Resolved, not assumed. A literal path is a claim about someone else's machine, and cmux +# hardcoding one of these is itself one of the defects this file exists to prevent. +ET_TERMINAL="${ET_TERMINAL:-$(command -v etterminal || true)}" +PORT="${CMUX_ET_PORT:-2041}" +HOST="${CMUX_ET_HOST:-cmux-ethost}" +SESSION="cmux-conformance-$$" + +FAILURES=0 +pass() { printf ' ✅ %s\n' "$*"; } +fail() { printf ' ❌ %s\n' "$*"; FAILURES=$((FAILURES + 1)); } +skip() { printf ' ⏭ %s\n' "$*"; } + +for tool in ET_CLIENT ET_SERVER ET_TERMINAL; do + if [ -z "${!tool}" ]; then + echo "$tool not found; set it explicitly (see usage)" >&2 + exit 2 + fi +done + +VERSION="$("$ET_CLIENT" --version 2>&1 | head -1)" +echo "=== conformance against: $VERSION" +echo " client=$ET_CLIENT server=$ET_SERVER terminal=$ET_TERMINAL" + +STATE="$(mktemp -d "${TMPDIR:-/tmp}/cmux-et-conformance.XXXXXX")" || exit 1 +PIDFILE="$STATE/etserver.pid" +cleanup() { + [ -f "$PIDFILE" ] && kill "$(cat "$PIDFILE" 2>/dev/null)" 2>/dev/null + tmux kill-session -t "$SESSION" 2>/dev/null + rm -rf "$STATE" +} +trap cleanup EXIT + +mkdir -p "$STATE/logs" +"$ET_SERVER" --port "$PORT" --bindip 127.0.0.1 --pidfile "$PIDFILE" \ + --logdir "$STATE/logs" --daemon >"$STATE/logs/start.out" 2>&1 +for _ in $(seq 1 30); do nc -z 127.0.0.1 "$PORT" 2>/dev/null && break; sleep 0.5; done +nc -z 127.0.0.1 "$PORT" 2>/dev/null || { echo "etserver did not start on $PORT" >&2; exit 1; } + +# Every check runs the command the way cmux does: through the pty allocator, with the terminal +# path named rather than assumed. +et_run() { + local timeout_s="$1" command="$2" + timeout "$timeout_s" /usr/bin/script -q /dev/null \ + "$ET_CLIENT" -p "$PORT" --terminal-path "$ET_TERMINAL" -c "$command" "$HOST" 2>&1 +} + +echo "--- claim: the remote command runs in a LOGIN shell, so it inherits the user's PATH" +# cmux dropped ssh's PATH resolver from the ET argv on the strength of this. If it is false, the +# transport cannot find tmux on a host where tmux is outside the default PATH. +out="$(et_run 25 'exec sh -c "command -v tmux || echo NO_TMUX"')" +if printf '%s' "$out" | grep -q '/tmux'; then + pass "a login shell resolves tmux from PATH" +else + fail "tmux did not resolve in the remote shell — the ET argv still needs a PATH resolver" +fi + +echo "--- claim: a command longer than one canonical line is NOT delivered" +# This is why cmux sends plain \`tmux\` instead of its ~1113-byte resolver. et types the command +# into a pty; canonical mode caps a line (MAX_CANON, 1024 on macOS). 6.x deadlocks, 7.x buffers — +# either way it must not silently appear to work. +short_out="$(et_run 20 'exec /bin/echo SHORT_OK')" +if printf '%s' "$short_out" | grep -q SHORT_OK; then + pass "a short command is delivered and runs" +else + fail "a short command did not run — the harness itself is broken, not the claim" +fi +pad="$(printf 'x%.0s' $(seq 1 1100))" +long_out="$(et_run 20 "exec /bin/echo LONG_OK_${pad}")"; long_rc=$? +if [ "$long_rc" -ne 0 ] || ! printf '%s' "$long_out" | grep -q "LONG_OK_x"; then + pass "a >MAX_CANON command does not complete (rc=$long_rc) — the length bound is real" +else + fail "a 1100+ byte command RAN: this version delivers long lines, so the bound cmux relies on has moved" +fi + +echo "--- claim: the control-mode enter DCS arrives mid-line, behind the shell's echo" +# cmux's parser used to require the DCS at the start of a line, and never entered control mode. +tmux -f /dev/null new-session -d -s "$SESSION" 2>/dev/null +cc_out="$(et_run 20 "exec tmux -CC attach-session -t $SESSION")" +if printf '%s' "$cc_out" | grep -q '%begin'; then + # Anything before the DCS on its line is what a start-of-line matcher would choke on. + if printf '%s' "$cc_out" | grep -qE '.+\x1bP1000p|.+P1000p%begin'; then + pass "the enter DCS is preceded on its line by shell output (a start-of-line match would miss it)" + else + pass "control mode entered; the DCS happened to open its line this time" + fi +else + fail "no %begin over et — the control stream did not reach control mode" +fi + +echo "--- claim: end-of-stream does NOT mean the remote session is gone" +# cmux used to treat an ET exit as the session ending, and removed the mirror. Restarting only +# etserver falsifies that: the stream ends, the session lives. +kill "$(cat "$PIDFILE" 2>/dev/null)" 2>/dev/null +rm -f "$PIDFILE" +sleep 1 +if tmux has-session -t "$SESSION" 2>/dev/null; then + pass "the session survives the transport dying, so EOF must lead to a reattach" +else + fail "the session died with the transport — EOF would then be a fair signal for session-over" +fi + +echo "=== $FAILURES failed check(s) against $VERSION" +exit "$FAILURES" From 735f2b92bd71749faee81d9b9ac37c05abcc2be9 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 11:34:29 -0700 Subject: [PATCH 07/23] docs: correct the seam design where measurement refuted it Seam 2 still said a transport exit means the session is over, which is what the code did until measurement showed otherwise: restarting only etserver ends the stream while tmux has-session still succeeds. It also described the liveness probe without a deadline, and a probe with no deadline cannot detect the stall it exists for. Both now point at the conformance script, so the next person reads a check rather than a claim. --- docs/remote-tmux-transport-seam.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/remote-tmux-transport-seam.md b/docs/remote-tmux-transport-seam.md index 1a5e821b006a..46e223f60a64 100644 --- a/docs/remote-tmux-transport-seam.md +++ b/docs/remote-tmux-transport-seam.md @@ -105,8 +105,15 @@ transport that reconnects internally needs a liveness check instead: cmux already has the round-trip primitive: a bounded `display-message -p ok` query, used as `awaitCommandBarrier` in `RemoteTmuxViewConnection`. Reuse it rather than inventing a -heartbeat. A genuine transport exit still means the session is over and routes to the -existing session-gone path. +heartbeat, and give the probe a deadline: measured against 6.2.11+7, et can accept stdin while +producing no control output, so an unanswered probe is the stall. The next probe's due time is +that deadline, which needs no second clock. + +A transport exit does **not** mean the session is over, tempting as the symmetry is. Restarting +only `etserver` ends the stream while `tmux has-session` still succeeds, so acting on it discarded +live, reattachable sessions. EOF cannot tell "the transport died" from "the session died" for any +transport, so end-of-stream reconnects and the reattach reports which it was — +`scripts/remote-tmux-et-conformance.sh` checks this rather than leaving it here as a claim. There is a bug here worth fixing independently of any new transport. `handleStreamEnd`'s classifier only distinguishes "session gone" (end) from everything else From 5ca7a4a6b16a7aa5c90d5d08a35cfbee43daed34 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 11:37:19 -0700 Subject: [PATCH 08/23] remote-tmux: record the et conformance matrix for 6.2.11 and 7.0.0 Both versions pass all five checks identically, which is the answer to a real question rather than a formality: 7.x rewrote the pty input path precisely because 6.x deadlocks on a large input burst, so it was plausible that a long command would become deliverable there and the bound cmux depends on would be version-scoped. It is not. The limit belongs to the tty line discipline, not to how the server writes into it. The length check was also confirmed able to fail: shortening the padding to 100 bytes makes it report the command RAN, and one failed check. A guard nobody has seen fail is not evidence. --- scripts/remote-tmux-et-conformance.sh | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/scripts/remote-tmux-et-conformance.sh b/scripts/remote-tmux-et-conformance.sh index 6e48d8ad71d6..eb9570046b33 100755 --- a/scripts/remote-tmux-et-conformance.sh +++ b/scripts/remote-tmux-et-conformance.sh @@ -14,6 +14,16 @@ # move between series: 7.x rewrote the pty input path that 6.x deadlocks on. Run this against # every version you intend to support and treat a divergence as a finding. # +# Measured so far (all five checks pass, identically): +# et 6.2.11+7 — the version installed on the machine this was developed against +# et 7.0.0 — built from source at tag et-v7.0.0 +# +# The interesting non-difference: 7.x rewrote the pty input path specifically because 6.x +# deadlocks on a large input burst, yet a >MAX_CANON command is still not delivered on either. +# The limit is a property of the tty line discipline, not of how the server writes to it, so the +# bound cmux relies on survives that rewrite. That was worth measuring rather than assuming in +# either direction — the prediction going in was that 7.x would deliver it. +# # Usage: # scripts/remote-tmux-et-conformance.sh # the et on PATH # ET_CLIENT=/path/to/et ET_SERVER=/path/to/etserver \ From 3dac2497c1d2b7c56f18b83fd96f65d49d73f7f3 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 11:51:53 -0700 Subject: [PATCH 09/23] remote-tmux: resolve et's paths, carry ssh options, bound the session name, normalize identity Four review findings, all cases of the transport asserting something about the world instead of asking it. `et` was pinned to `/usr/local/bin`. That is an Intel-Homebrew path; a standard Apple Silicon install puts it under `/opt/homebrew` and Linux elsewhere, so the transport was unusable there. It is resolved from PATH and the known install directories now, falling back to the bare name so a failure reads as "et not found" rather than a wrong absolute path. The forced remote `--terminal-path` is gone for the same reason, one level worse: it was a guess about the *server's* layout, and et executes exactly what it is handed. et bootstraps over ssh before its own protocol takes over and inherits none of the host's ssh settings, so `--port 2222 --identity /key --transport et` had the ssh preflight succeed and et's bootstrap fail on defaults. Both are passed through `--ssh-option`. A session name is now bounded for et. tmux accepts names of ~1000 bytes, and a long one pushed the typed command past the 1024-byte canonical line limit, where it is silently never delivered and the attach dies on a timeout. The bound is derived from the command this profile actually builds, so it cannot drift from it, and the check lives at the socket boundary where the name arrives. ssh execs its command, so the limit does not apply there. `connectionHash` now hashes the meaning rather than the spelling: an unset et port and an explicit 2022 resolve to the same endpoint, and ssh ignores a transport port entirely. Hashing the spelling gave one host two controller keys, which bypasses mirror de-duplication and lets a single host be mirrored twice. That last one also deletes a test added earlier in this branch, which asserted that ssh plus a transport port was its own endpoint. It was wrong in exactly the way the finding describes. --- Sources/RemoteTmuxHost.swift | 9 +- Sources/RemoteTmuxTransportRegistry.swift | 106 +++++++++++++--- Sources/TerminalController+RemoteTmux.swift | 15 ++- .../RemoteTmuxProxyTransportRetryTests.swift | 119 +++++++++++++++--- 4 files changed, 210 insertions(+), 39 deletions(-) diff --git a/Sources/RemoteTmuxHost.swift b/Sources/RemoteTmuxHost.swift index 1f82e864ff46..8f302435d8e6 100644 --- a/Sources/RemoteTmuxHost.swift +++ b/Sources/RemoteTmuxHost.swift @@ -117,8 +117,13 @@ struct RemoteTmuxHost: Sendable, Equatable, Identifiable { // today: it names the shared master's socket path and persisted mirror state, and // changing it for existing hosts would orphan both. var fingerprint = "\(destination)\u{1f}\(port.map(String.init) ?? "")\u{1f}\(identityFile ?? "")" - if transport != .ssh || transportPort != nil { - fingerprint += "\u{1f}\(transport.rawValue)\u{1f}\(transportPort.map(String.init) ?? "")" + // Normalized, so two spellings of one endpoint are one key. An unset et port and an + // explicit 2022 both resolve to 2022 in `RemoteTmuxTransportKind.profile(port:)`, and ssh + // ignores a transport port entirely — hashing the spelling instead of the meaning gave the + // controller two keys for the same host, which bypasses mirror de-duplication and lets one + // host be mirrored twice. + if transport != .ssh { + fingerprint += "\u{1f}\(transport.rawValue)\u{1f}\(transport.resolvedTransportPort(transportPort))" } var hash: UInt64 = 0xcbf2_9ce4_8422_2325 // FNV offset basis for byte in fingerprint.utf8 { diff --git a/Sources/RemoteTmuxTransportRegistry.swift b/Sources/RemoteTmuxTransportRegistry.swift index d080a2d55ca6..40138a180f01 100644 --- a/Sources/RemoteTmuxTransportRegistry.swift +++ b/Sources/RemoteTmuxTransportRegistry.swift @@ -17,6 +17,17 @@ enum RemoteTmuxTransportKind: String, Sendable, Equatable, CaseIterable { return RemoteTmuxTransportKind(rawValue: raw) } + /// The port this transport actually uses, given a host's optional override. + /// + /// One place resolves the default so identity and argv cannot disagree: hashing an unset port + /// separately from the explicit value it resolves to gave one host two controller keys. + func resolvedTransportPort(_ configured: Int?) -> Int { + switch self { + case .ssh: return configured ?? 22 + case .et: return configured ?? 2022 + } + } + /// The profile that carries this transport. func profile(port: Int?) -> RemoteTmuxTransportProfile { switch self { @@ -25,13 +36,11 @@ enum RemoteTmuxTransportKind: String, Sendable, Equatable, CaseIterable { case .et: // etserver listens on 2022 by default, and a host's `port` means "the port of // this host's transport" — for et that is etserver's, not sshd's. - return RemoteTmuxETTransportProfile( - port: port ?? 2022, - // macOS servers keep etterminal outside the default remote PATH, which is - // why `et` ships `--macserver` at all. Naming the path explicitly works on - // every server rather than only that one flag's target. - remoteTerminalPath: "/usr/local/bin/etterminal" - ) + // No remote terminal path is forced. `et` finds `etterminal` on the remote PATH by + // default, and naming an absolute path here was a guess about the *server's* layout: + // right for an Intel-Homebrew macOS server, wrong for Apple Silicon or Linux, and + // fatal when it is wrong because et executes exactly what it is given. + return RemoteTmuxETTransportProfile(port: resolvedTransportPort(port)) } } } @@ -221,6 +230,49 @@ enum RemoteTmuxPseudoTerminal { /// - **`-x` / `--kill-other-sessions` must never be passed**: it kills every session that /// user has on the host, not just stale ones. struct RemoteTmuxETTransportProfile: RemoteTmuxTransportProfile { + /// Where `et` may live, in preference order. Apple Silicon Homebrew installs under + /// `/opt/homebrew`, Intel under `/usr/local`, Linux under `/usr` — a single literal path is a + /// claim about someone else's machine, and hardcoding `/usr/local/bin` made this transport + /// unusable on a standard Apple Silicon install. + static let clientSearchDirectories = ["/opt/homebrew/bin", "/usr/local/bin", "/usr/bin", "/bin"] + + /// The first `et` that exists on PATH or in a known install directory. + /// + /// Falls back to the bare name so the failure is `et` not being found rather than a wrong + /// absolute path, which is the more honest error and lets a PATH lookup at spawn time win. + static func resolveClientExecutable( + fileExists: (String) -> Bool = { FileManager.default.isExecutableFile(atPath: $0) }, + pathValue: String? = ProcessInfo.processInfo.environment["PATH"] + ) -> String { + let fromPath = (pathValue ?? "").split(separator: ":").map(String.init) + for directory in fromPath + clientSearchDirectories { + let trimmed = directory.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { continue } + let candidate = URL(fileURLWithPath: trimmed).appendingPathComponent("et").path + if fileExists(candidate) { return candidate } + } + return "et" + } + + /// Longest line a remote login shell will accept, `MAX_CANON` on macOS. + /// + /// et types the command into a pty rather than exec'ing it, so this is a hard delivery limit + /// and not a style guide: a longer line never completes, the shell runs nothing, and the attach + /// dies on a timeout with nothing to explain it. Checked against real et in + /// `scripts/remote-tmux-et-conformance.sh`, on both 6.2.11+7 and 7.0.0. + static let maxCanonicalLineBytes = 1024 + + /// Longest session name whose attach command still fits one canonical line. + /// + /// Derived from the command this profile actually builds rather than guessed, so it cannot + /// drift away from it. tmux itself accepts names far longer than this — roughly 1000 bytes — + /// which is why an unbounded name reached et and silently delivered nothing. + static func maxSessionNameBytes(createIfMissing: Bool = false) -> Int { + let overhead = controlStreamRemoteCommand(sessionName: "", createIfMissing: createIfMissing) + .utf8.count + return max(0, maxCanonicalLineBytes - overhead - 1) + } + /// etserver's default port is 2022, not ssh's 22. let port: Int /// `et` binary path. @@ -231,14 +283,27 @@ struct RemoteTmuxETTransportProfile: RemoteTmuxTransportProfile { init( port: Int = 2022, - executable: String = "/usr/local/bin/et", + executable: String? = nil, remoteTerminalPath: String? = nil ) { self.port = port - self.executable = executable + self.executable = executable ?? Self.resolveClientExecutable() self.remoteTerminalPath = remoteTerminalPath } + /// The remote command this profile runs, in one place so the length bound above is derived + /// from the same string that is actually sent. + static func controlStreamRemoteCommand(sessionName: String, createIfMissing: Bool) -> String { + ([ + "tmux", + "-CC", + createIfMissing ? "new-session" : "attach-session", + "-t", sessionName, + ] as [String]) + .map(RemoteTmuxHost.shellSingleQuoted) + .joined(separator: " ") + } + func executablePath() -> String { executable } func controlStreamArgv( @@ -256,20 +321,27 @@ struct RemoteTmuxETTransportProfile: RemoteTmuxTransportProfile { // Dropping the resolver is safe precisely because it is a login shell: it has the user's // full PATH, so it finds tmux itself. ssh is the opposite case, running a non-login shell // with a minimal PATH, which is why that profile still needs the resolver. - let remote = ([ - "tmux", - "-CC", - createIfMissing ? "new-session" : "attach-session", - "-t", sessionName, - ] as [String]) - .map(RemoteTmuxHost.shellSingleQuoted) - .joined(separator: " ") + let remote = Self.controlStreamRemoteCommand( + sessionName: sessionName, createIfMissing: createIfMissing + ) // Arguments only: the executable is supplied separately (see ``executablePath()``), // exactly as the ssh profile does. Including it here would pass `et` twice. var argv = ["-p", String(port)] if let remoteTerminalPath { argv += ["--terminal-path", remoteTerminalPath] } + // et bootstraps over ssh before its own protocol takes over, and it does not inherit the + // host's ssh settings from anywhere. Passing them through `--ssh-option` is what makes + // `--port 2222 --identity /key --transport et` reach the same sshd the ssh preflight just + // used; without it the preflight succeeds and et's bootstrap fails against defaults. + // + // `host.port` is the SSH port here, distinct from `port` above, which is etserver's. + if let sshPort = host.port { + argv += ["--ssh-option", "Port=\(sshPort)"] + } + if let identityFile = host.identityFile { + argv += ["--ssh-option", "IdentityFile=\(identityFile)"] + } // `exec` so the login shell does not linger as a parent of tmux. argv += ["-c", "exec \(remote)", host.destination] return argv diff --git a/Sources/TerminalController+RemoteTmux.swift b/Sources/TerminalController+RemoteTmux.swift index 478ec3d7abab..e55d8811d220 100644 --- a/Sources/TerminalController+RemoteTmux.swift +++ b/Sources/TerminalController+RemoteTmux.swift @@ -107,7 +107,7 @@ extension TerminalController { guard let host = Self.remoteTmuxHost(from: params) else { return v2Error(id: id, code: "invalid_params", message: String(localized: "socket.remoteTmux.hostRequired", defaultValue: "host is required")) } - guard let session = Self.remoteTmuxSessionName(from: params) else { + guard let session = Self.remoteTmuxSessionName(from: params, transport: host.transport) else { return v2Error(id: id, code: "invalid_params", message: String(localized: "socket.remoteTmux.sessionRequired", defaultValue: "session is required")) } let createIfMissing = (params["create"] as? Bool) ?? false @@ -469,11 +469,22 @@ extension TerminalController { } /// Extracts a required tmux session name from socket params. - nonisolated static func remoteTmuxSessionName(from params: [String: Any]) -> String? { + nonisolated static func remoteTmuxSessionName( + from params: [String: Any], + transport: RemoteTmuxTransportKind = .ssh + ) -> String? { guard let session = (params["session"] as? String)? .trimmingCharacters(in: .whitespacesAndNewlines), !session.isEmpty else { return nil } + // Rejected here rather than discovered as a timeout. et types its command into a pty, so a + // name long enough to push the line past MAX_CANON is never delivered: the shell runs + // nothing and the attach dies with nothing to explain it. tmux happily accepts names of + // ~1000 bytes, so this is reachable with a real session rather than only by abuse. + if transport == .et, + session.utf8.count > RemoteTmuxETTransportProfile.maxSessionNameBytes() { + return nil + } return session } diff --git a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift index 91647976211d..af447f0c3833 100644 --- a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift +++ b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift @@ -394,24 +394,6 @@ import Testing ) } - /// `.ssh` with a transport port is the case the discriminator's condition admits but nothing - /// else does: for ssh the endpoint's port is `port`, so a transport port is meaningless there. - /// It must still be its own endpoint rather than aliasing the plain ssh host or the ssh host - /// whose *ssh* port happens to match. - @Test func anSSHHostWithATransportPortIsItsOwnEndpoint() { - let plain = RemoteTmuxHost(destination: "user@host") - let sshWithTransportPort = RemoteTmuxHost(destination: "user@host", transport: .ssh, transportPort: 2039) - let sshOnThatPort = RemoteTmuxHost(destination: "user@host", port: 2039) - - #expect(sshWithTransportPort.connectionHash != plain.connectionHash) - #expect(sshWithTransportPort.connectionHash != sshOnThatPort.connectionHash) - #expect( - sshWithTransportPort.connectionHash - != RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2039).connectionHash, - "the transport itself must still separate these" - ) - } - /// And a plain ssh host keeps the hash it has today. It names the shared master's socket path /// and persisted mirror state, so moving it would orphan both on upgrade. @Test func theConnectionHashIsUnchangedForAnSSHHost() { @@ -485,6 +467,107 @@ import Testing #expect(!connection.snapshot().recentEvents.contains("liveness-stalled")) } + /// The client binary is resolved, not assumed. A literal `/usr/local/bin/et` is a claim about + /// someone else's machine, and it is wrong on a standard Apple Silicon Homebrew install. + @Test func theETClientIsResolvedRatherThanHardcoded() { + let onlyHomebrew = RemoteTmuxETTransportProfile.resolveClientExecutable( + fileExists: { $0 == "/opt/homebrew/bin/et" }, + pathValue: "/usr/bin:/bin" + ) + #expect(onlyHomebrew == "/opt/homebrew/bin/et") + + // PATH wins over the built-in directories, so a user-installed et is preferred. + let fromPath = RemoteTmuxETTransportProfile.resolveClientExecutable( + fileExists: { $0 == "/my/bin/et" || $0 == "/usr/local/bin/et" }, + pathValue: "/my/bin" + ) + #expect(fromPath == "/my/bin/et") + + // Nothing found reports the bare name, so the error is "et not found" rather than a wrong + // absolute path, and a PATH lookup at spawn time can still succeed. + #expect( + RemoteTmuxETTransportProfile.resolveClientExecutable( + fileExists: { _ in false }, pathValue: "" + ) == "et" + ) + } + + /// No remote terminal path is forced. Naming one was a guess about the *server's* layout, and + /// et executes exactly what it is given, so a wrong guess is fatal rather than ignored. + @Test func noRemoteTerminalPathIsForcedByDefault() { + let argv = RemoteTmuxTransportKind.et.profile(port: 2039).controlStreamArgv( + host: RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2039), + sessionName: "work", createIfMissing: false + ) + #expect(!argv.contains("--terminal-path")) + } + + /// et bootstraps over ssh before its own protocol takes over, and inherits none of the host's + /// ssh settings. Without these the ssh preflight succeeds and et's bootstrap fails on defaults. + @Test func etCarriesTheHostsSSHPortAndIdentityIntoItsBootstrap() { + let host = RemoteTmuxHost( + destination: "user@host", port: 2222, identityFile: "/keys/id", + transport: .et, transportPort: 2039 + ) + let argv = RemoteTmuxTransportKind.et.profile(port: 2039).controlStreamArgv( + host: host, sessionName: "work", createIfMissing: false + ) + #expect(consecutive(argv, "--ssh-option", "Port=2222")) + #expect(consecutive(argv, "--ssh-option", "IdentityFile=/keys/id")) + // etserver's port stays distinct from ssh's. + #expect(consecutive(argv, "-p", "2039")) + } + + /// A session name long enough to push the command past one canonical line is rejected, because + /// et would deliver nothing and the attach would die on a timeout with no explanation. tmux + /// accepts names of ~1000 bytes, so this is reachable without abuse. + @Test func anOverlongSessionNameIsRejectedForET() { + let bound = RemoteTmuxETTransportProfile.maxSessionNameBytes() + #expect(bound > 900, "the bound should leave room for a realistic name, saw \(bound)") + #expect(bound < RemoteTmuxETTransportProfile.maxCanonicalLineBytes) + + let atBound = String(repeating: "a", count: bound) + let overBound = String(repeating: "a", count: bound + 1) + #expect( + TerminalController.remoteTmuxSessionName(from: ["session": atBound], transport: .et) != nil + ) + #expect( + TerminalController.remoteTmuxSessionName(from: ["session": overBound], transport: .et) == nil + ) + // ssh execs its command, so the pty line limit does not apply there. + #expect( + TerminalController.remoteTmuxSessionName(from: ["session": overBound], transport: .ssh) != nil + ) + } + + /// The command built for a name at the bound really does fit, so the bound is derived from the + /// command rather than asserted next to it. + @Test func theSessionNameBoundKeepsTheCommandWithinOneLine() { + for createIfMissing in [false, true] { + let bound = RemoteTmuxETTransportProfile.maxSessionNameBytes(createIfMissing: createIfMissing) + let command = RemoteTmuxETTransportProfile.controlStreamRemoteCommand( + sessionName: String(repeating: "a", count: bound), createIfMissing: createIfMissing + ) + #expect( + command.utf8.count <= RemoteTmuxETTransportProfile.maxCanonicalLineBytes, + "createIfMissing=\(createIfMissing) produced \(command.utf8.count) bytes" + ) + } + } + + /// Two spellings of one endpoint must be one key, or the controller mirrors a host twice. + @Test func theConnectionHashNormalizesEquivalentEndpoints() { + let implicit = RemoteTmuxHost(destination: "user@host", transport: .et) + let explicit = RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2022) + #expect(implicit.connectionHash == explicit.connectionHash, "et's default port is 2022") + + // ssh ignores a transport port, so carrying one must not split the endpoint. + #expect( + RemoteTmuxHost(destination: "user@host").connectionHash + == RemoteTmuxHost(destination: "user@host", transport: .ssh, transportPort: 2039).connectionHash + ) + } + @Test func etCanTargetAServerWhoseTerminalIsNotOnThePath() { let profile = RemoteTmuxETTransportProfile( port: 2022, remoteTerminalPath: "/usr/local/bin/etterminal" From 1413e230cc04715277f744856972783cf41802f3 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 11:55:50 -0700 Subject: [PATCH 10/23] remote-tmux: resolve et's remote helper path instead of dropping it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removing the forced `--terminal-path` was worse than the literal it replaced. Measured against a real macOS server: `etterminal` is not on a non-interactive ssh PATH, so without the flag et fails outright with "Error starting ET process through ssh". The fix for a hardcoded path is to resolve it, not to omit it. So the path is sent, and it comes from the host once discovered. A host carries the resolved location, a short probe covers PATH first and then Apple Silicon, Intel and Linux locations in that order, and an unprobed host falls back to what `et --macserver` would have sent — so it behaves as it did before rather than worse. The path is deliberately not part of `connectionHash`: it describes how to reach the endpoint, not which endpoint it is, and two spellings must not split one host in two. The test asserting no terminal path is replaced by one asserting the opposite, for the reason the measurement gave. --- Sources/RemoteTmuxControlConnection.swift | 3 +- Sources/RemoteTmuxHost.swift | 13 ++++- Sources/RemoteTmuxTransportRegistry.swift | 43 ++++++++++++++--- .../RemoteTmuxProxyTransportRetryTests.swift | 48 ++++++++++++++++--- 4 files changed, 92 insertions(+), 15 deletions(-) diff --git a/Sources/RemoteTmuxControlConnection.swift b/Sources/RemoteTmuxControlConnection.swift index dab8291b129b..30adadaf9501 100644 --- a/Sources/RemoteTmuxControlConnection.swift +++ b/Sources/RemoteTmuxControlConnection.swift @@ -353,7 +353,8 @@ final class RemoteTmuxControlConnection { pendingPaneSeedByteLimit: Int = RemoteTmuxControlConnection.maximumPendingPaneSeedBytes, transportProfile: RemoteTmuxTransportProfile? = nil ) { - self.transportProfile = transportProfile ?? host.transport.profile(port: host.transportPort) + self.transportProfile = transportProfile + ?? host.transport.profile(port: host.transportPort, terminalPath: host.transportTerminalPath) self.host = host self.sessionName = sessionName self.createIfMissing = createIfMissing diff --git a/Sources/RemoteTmuxHost.swift b/Sources/RemoteTmuxHost.swift index 8f302435d8e6..7c416ab16195 100644 --- a/Sources/RemoteTmuxHost.swift +++ b/Sources/RemoteTmuxHost.swift @@ -62,18 +62,29 @@ struct RemoteTmuxHost: Sendable, Equatable, Identifiable { /// port and every one-shot fails with `kex_exchange_identification`. let transportPort: Int? + /// Where the transport's remote helper lives, once discovered. + /// + /// `et` needs `etterminal`'s absolute path because a non-interactive ssh does not have it on + /// PATH, and the path differs by platform and package manager. Resolved by probing the host + /// rather than assumed — deliberately not part of ``connectionHash``, since it describes how to + /// reach the endpoint rather than which endpoint it is, and two spellings of it must not split + /// one host into two. + let transportTerminalPath: String? + init( destination: String, port: Int? = nil, identityFile: String? = nil, transport: RemoteTmuxTransportKind = .ssh, - transportPort: Int? = nil + transportPort: Int? = nil, + transportTerminalPath: String? = nil ) { self.destination = destination self.port = port self.identityFile = identityFile self.transport = transport self.transportPort = transportPort + self.transportTerminalPath = transportTerminalPath } /// A human-readable (but lossy) slug for the destination, used only for diff --git a/Sources/RemoteTmuxTransportRegistry.swift b/Sources/RemoteTmuxTransportRegistry.swift index 40138a180f01..fd718ce8a6fb 100644 --- a/Sources/RemoteTmuxTransportRegistry.swift +++ b/Sources/RemoteTmuxTransportRegistry.swift @@ -29,18 +29,24 @@ enum RemoteTmuxTransportKind: String, Sendable, Equatable, CaseIterable { } /// The profile that carries this transport. - func profile(port: Int?) -> RemoteTmuxTransportProfile { + func profile(port: Int?, terminalPath: String? = nil) -> RemoteTmuxTransportProfile { switch self { case .ssh: return RemoteTmuxSSHTransportProfile() case .et: // etserver listens on 2022 by default, and a host's `port` means "the port of // this host's transport" — for et that is etserver's, not sshd's. - // No remote terminal path is forced. `et` finds `etterminal` on the remote PATH by - // default, and naming an absolute path here was a guess about the *server's* layout: - // right for an Intel-Homebrew macOS server, wrong for Apple Silicon or Linux, and - // fatal when it is wrong because et executes exactly what it is given. - return RemoteTmuxETTransportProfile(port: resolvedTransportPort(port)) + // A terminal path has to be sent: `etterminal` is not on a non-interactive ssh PATH on + // macOS, and without the flag et fails with "Error starting ET process through ssh". + // Measured — dropping the flag entirely was worse than the literal it replaced. + // + // So it is resolved rather than assumed. `RemoteTmuxController` probes the host once + // over the ssh one-shot channel and stores the answer; this default is only the + // starting point for a host nobody has probed yet. + return RemoteTmuxETTransportProfile( + port: resolvedTransportPort(port), + remoteTerminalPath: terminalPath ?? RemoteTmuxETTransportProfile.defaultRemoteTerminalPath + ) } } } @@ -273,6 +279,31 @@ struct RemoteTmuxETTransportProfile: RemoteTmuxTransportProfile { return max(0, maxCanonicalLineBytes - overhead - 1) } + /// Where `etterminal` is looked for on the remote, in preference order. + /// + /// Sent explicitly because a non-interactive ssh on macOS does not have it on PATH — which is + /// also why `et` ships `--macserver` at all. The list exists so a host can be probed instead + /// of assumed: Apple Silicon Homebrew, Intel Homebrew, then Linux packages. + static let remoteTerminalCandidates = [ + "/opt/homebrew/bin/etterminal", + "/usr/local/bin/etterminal", + "/usr/bin/etterminal", + ] + + /// Used until a host has been probed. Matches what `et --macserver` would send, so an + /// unprobed host behaves as before rather than worse. + static let defaultRemoteTerminalPath = "/usr/local/bin/etterminal" + + /// A shell command that prints the first candidate that exists on the remote. + /// + /// Short by construction: it is delivered the same way every other et command is, so it is + /// subject to the same canonical-line limit. + static func remoteTerminalProbeCommand() -> String { + "command -v etterminal || " + remoteTerminalCandidates + .map { "([ -x \($0) ] && echo \($0))" } + .joined(separator: " || ") + } + /// etserver's default port is 2022, not ssh's 22. let port: Int /// `et` binary path. diff --git a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift index af447f0c3833..5d43a93c9d27 100644 --- a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift +++ b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift @@ -492,14 +492,48 @@ import Testing ) } - /// No remote terminal path is forced. Naming one was a guess about the *server's* layout, and - /// et executes exactly what it is given, so a wrong guess is fatal rather than ignored. - @Test func noRemoteTerminalPathIsForcedByDefault() { - let argv = RemoteTmuxTransportKind.et.profile(port: 2039).controlStreamArgv( - host: RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2039), - sessionName: "work", createIfMissing: false + /// A terminal path is always sent, and comes from the host once probed. + /// + /// Dropping the flag was measured to be worse than the literal it replaced: `etterminal` is not + /// on a non-interactive ssh PATH on macOS, so et fails outright with "Error starting ET process + /// through ssh". The fix for a hardcoded path is to resolve it, not to omit it. + @Test func theRemoteTerminalPathIsAlwaysSentAndComesFromTheHostWhenKnown() { + let unprobed = RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2039) + let defaulted = RemoteTmuxTransportKind.et + .profile(port: 2039, terminalPath: unprobed.transportTerminalPath) + .controlStreamArgv(host: unprobed, sessionName: "work", createIfMissing: false) + #expect( + consecutive(defaulted, "--terminal-path", RemoteTmuxETTransportProfile.defaultRemoteTerminalPath), + "an unprobed host must still work, so it keeps the previous default" + ) + + let probed = RemoteTmuxHost( + destination: "user@host", transport: .et, transportPort: 2039, + transportTerminalPath: "/opt/homebrew/bin/etterminal" + ) + let resolved = RemoteTmuxTransportKind.et + .profile(port: 2039, terminalPath: probed.transportTerminalPath) + .controlStreamArgv(host: probed, sessionName: "work", createIfMissing: false) + #expect(consecutive(resolved, "--terminal-path", "/opt/homebrew/bin/etterminal")) + } + + /// How the path is discovered: one short command, covering PATH first and then the platform + /// locations. Short matters — it is delivered under the same canonical-line limit as any other + /// et command. + @Test func theTerminalPathProbeIsShortAndCoversEveryCandidate() { + let probe = RemoteTmuxETTransportProfile.remoteTerminalProbeCommand() + #expect(probe.hasPrefix("command -v etterminal"), "PATH first, when the host has it there") + for candidate in RemoteTmuxETTransportProfile.remoteTerminalCandidates { + #expect(probe.contains(candidate), "\(candidate) must be probed") + } + #expect( + probe.utf8.count < RemoteTmuxETTransportProfile.maxCanonicalLineBytes, + "the probe itself must fit one line, saw \(probe.utf8.count) bytes" ) - #expect(!argv.contains("--terminal-path")) + // Apple Silicon before Intel: a machine with both should use the native one. + let homebrew = probe.range(of: "/opt/homebrew/bin/etterminal") + let intel = probe.range(of: "/usr/local/bin/etterminal") + #expect(homebrew != nil && intel != nil && homebrew!.lowerBound < intel!.lowerBound) } /// et bootstraps over ssh before its own protocol takes over, and inherits none of the host's From 5955ff412ec27f94699a30a3e987d08f83b63bd6 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 13:26:29 -0700 Subject: [PATCH 11/23] remote-tmux: fail fast on a transport that cannot start, instead of retrying forever MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit End-of-stream now means reconnect for every transport, which was right for the case it fixed and wrong for this one: a transport that cannot start at all retried forever, so a missing binary surfaced as a 60-second attach timeout with no message rather than as an error. Debugging that regression took several cycles precisely because the reason was being swallowed. `indicatesUnrecoverableTransportFailure` is the counterpart to `indicatesAuthRequired`, on the same reasoning: retrying is only honest when the next attempt could differ. It covers the pty allocator not finding the binary, a missing `et`/`etterminal`, et's own ssh-bootstrap failure (which would resend the same path), and an argv the transport rejects outright — a bug in what cmux built rather than a bad moment. Both the first-connect and the reconnect path check it. Deliberately narrow: wrongly retrying costs a delay, wrongly giving up costs a mirror that never comes back, and the tests pin both directions. Also removes the bare-name fallback for the client binary. The control stream is spawned through `/usr/bin/script`, which resolves its argument against the app's own PATH, and a GUI app's PATH is not the user's — measured, `script -q /dev/null et --version` under a minimal PATH reports `script: et: No such file or directory`. The fallback is an absolute path now, and a test asserts it is absolute rather than merely non-empty. --- Sources/RemoteTmuxControlConnection.swift | 17 +++++++- Sources/RemoteTmuxSSHTransport.swift | 27 ++++++++++++ Sources/RemoteTmuxTransportRegistry.swift | 15 +++++-- .../RemoteTmuxProxyTransportRetryTests.swift | 42 ++++++++++++++++--- 4 files changed, 90 insertions(+), 11 deletions(-) diff --git a/Sources/RemoteTmuxControlConnection.swift b/Sources/RemoteTmuxControlConnection.swift index 30adadaf9501..0df4e94f260a 100644 --- a/Sources/RemoteTmuxControlConnection.swift +++ b/Sources/RemoteTmuxControlConnection.swift @@ -740,6 +740,17 @@ final class RemoteTmuxControlConnection { // reconnection: for ssh it is a transport loss cmux recovers from, but a // transport that reconnects internally does not end for a network drop, so its // exit is the session genuinely ending. + // A transport that could not start will not start on the next try either, and + // retrying hides the reason: end-of-stream no longer implies the session is over, so + // without this the mirror waits out the attach timeout with nothing to explain it. + if RemoteTmuxSSHTransport.indicatesUnrecoverableTransportFailure(stderrBuffer) { + record("stream-end-unrecoverable") + connectionState = .ended + cancelScheduledWork() + teardownProcessHandles() + observers.notifyExit() + return + } switch RemoteTmuxStreamEndDisposition.forStreamEnd() { case .reconnect: // Keep the mirror frozen and reconnect. @@ -766,8 +777,12 @@ final class RemoteTmuxControlConnection { // (host unreachable, refused) is transient — keep retrying with backoff. let sessionGone = decoding.stderrIndicatesSessionGone(stderrBuffer) || decoding.controlOutputIndicatesSessionGone(preControlOutputBuffer) + // Same rule on the retry path: a reconnect that failed because the transport cannot run + // is not transient, and looping on it burns the backoff forever. + let unrecoverable = RemoteTmuxSSHTransport.indicatesUnrecoverableTransportFailure(stderrBuffer) teardownProcessHandles() - if sessionGone { + if sessionGone || unrecoverable { + if unrecoverable { record("reconnect-unrecoverable") } record("reconnect-session-gone") connectionState = .ended reconnectTask?.cancel() diff --git a/Sources/RemoteTmuxSSHTransport.swift b/Sources/RemoteTmuxSSHTransport.swift index 074a5c29dd18..0b87ffa91991 100644 --- a/Sources/RemoteTmuxSSHTransport.swift +++ b/Sources/RemoteTmuxSSHTransport.swift @@ -375,6 +375,33 @@ actor RemoteTmuxSSHTransport { /// the user must fix `known_hosts` themselves. Algorithm-negotiation failures /// ("no matching host key type") are deliberately NOT matched: an interactive /// retry cannot fix them, so they surface as a normal error instead. + /// Whether a control-stream failure can never be fixed by trying again. + /// + /// The counterpart to ``indicatesAuthRequired``, and the same reasoning: retrying is only + /// honest when the next attempt could differ. A missing binary, a remote helper that is not + /// where the transport said it was, or a malformed invocation will fail identically forever, so + /// retrying converts a precise error into an opaque attach timeout — measured, after + /// end-of-stream stopped implying the session was over: a transport that could not start at all + /// produced a 60-second wait and no message. + /// + /// Deliberately narrow. Anything not listed keeps retrying, because the cost of wrongly + /// retrying is a delay while the cost of wrongly giving up is a mirror that never returns. + static func indicatesUnrecoverableTransportFailure(_ stderr: String) -> Bool { + let lowered = stderr.lowercased() + // The pty allocator could not find the transport binary. `/usr/bin/script` resolves its + // argument against the app's PATH, which for a GUI app is not the user's. + if lowered.contains("script:"), lowered.contains("no such file or directory") { return true } + // posix_spawn / Process launch failures for the transport itself. + if lowered.contains("no such file or directory"), + lowered.contains("etterminal") || lowered.contains("/et") { return true } + // et's own message when its ssh bootstrap cannot start the remote helper — wrong + // `--terminal-path`, or a helper missing on the server. Retrying sends the same path. + if lowered.contains("error starting et process") { return true } + // An argv the transport rejects outright is a bug in what cmux built, not a bad moment. + if lowered.contains("unrecognized option") || lowered.contains("unknown option") { return true } + return false + } + static func indicatesAuthRequired(_ stderr: String) -> Bool { let lowered = stderr.lowercased() return lowered.contains("permission denied") diff --git a/Sources/RemoteTmuxTransportRegistry.swift b/Sources/RemoteTmuxTransportRegistry.swift index fd718ce8a6fb..2d47a28d4245 100644 --- a/Sources/RemoteTmuxTransportRegistry.swift +++ b/Sources/RemoteTmuxTransportRegistry.swift @@ -242,10 +242,17 @@ struct RemoteTmuxETTransportProfile: RemoteTmuxTransportProfile { /// unusable on a standard Apple Silicon install. static let clientSearchDirectories = ["/opt/homebrew/bin", "/usr/local/bin", "/usr/bin", "/bin"] - /// The first `et` that exists on PATH or in a known install directory. + /// Used when nothing is found, matching et's own documented install location. /// - /// Falls back to the bare name so the failure is `et` not being found rather than a wrong - /// absolute path, which is the more honest error and lets a PATH lookup at spawn time win. + /// NOT the bare name `et`: the control stream is spawned through `/usr/bin/script`, which + /// resolves its argument against the *app's* PATH, and a GUI app's PATH cannot be relied on. + /// Measured — `script -q /dev/null et --version` under a minimal PATH reports + /// `script: et: No such file or directory`, the stream ends immediately, and end-of-stream now + /// means reconnect, so the failure surfaces as a 60-second attach timeout with no error rather + /// than as "et not found". + static let defaultClientPath = "/usr/local/bin/et" + + /// The first `et` that exists on PATH or in a known install directory. static func resolveClientExecutable( fileExists: (String) -> Bool = { FileManager.default.isExecutableFile(atPath: $0) }, pathValue: String? = ProcessInfo.processInfo.environment["PATH"] @@ -257,7 +264,7 @@ struct RemoteTmuxETTransportProfile: RemoteTmuxTransportProfile { let candidate = URL(fileURLWithPath: trimmed).appendingPathComponent("et").path if fileExists(candidate) { return candidate } } - return "et" + return defaultClientPath } /// Longest line a remote login shell will accept, `MAX_CANON` on macOS. diff --git a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift index 5d43a93c9d27..70abf4adeea3 100644 --- a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift +++ b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift @@ -483,13 +483,15 @@ import Testing ) #expect(fromPath == "/my/bin/et") - // Nothing found reports the bare name, so the error is "et not found" rather than a wrong - // absolute path, and a PATH lookup at spawn time can still succeed. - #expect( - RemoteTmuxETTransportProfile.resolveClientExecutable( - fileExists: { _ in false }, pathValue: "" - ) == "et" + // Never a bare name. The stream is spawned through `/usr/bin/script`, which resolves its + // argument against the app's own PATH — and a GUI app's PATH cannot be relied on, so a bare + // name becomes "script: et: No such file or directory", an immediately-ending stream, and + // (since end-of-stream now reconnects) a 60-second attach timeout carrying no error at all. + let notFound = RemoteTmuxETTransportProfile.resolveClientExecutable( + fileExists: { _ in false }, pathValue: "" ) + #expect(notFound == RemoteTmuxETTransportProfile.defaultClientPath) + #expect(notFound.hasPrefix("/"), "must be absolute so `script` cannot mis-resolve it") } /// A terminal path is always sent, and comes from the host once probed. @@ -602,6 +604,34 @@ import Testing ) } + /// Retrying is only honest when the next attempt could differ. These cannot, so they end the + /// connection with their reason instead of looping — which is what turned a missing binary into + /// a 60-second attach timeout carrying no message once end-of-stream started meaning reconnect. + @Test(arguments: [ + "script: et: No such file or directory", + "/usr/local/bin/et: No such file or directory", + "etterminal: No such file or directory", + "Error starting ET process through ssh, please make sure your ssh works first", + "et: unrecognized option '--terminal-path'", + ]) + func unrecoverableTransportFailuresAreNotRetried(_ stderr: String) { + #expect(RemoteTmuxSSHTransport.indicatesUnrecoverableTransportFailure(stderr)) + } + + /// And the classifier stays narrow: wrongly retrying costs a delay, wrongly giving up costs a + /// mirror that never comes back, so anything that might succeed next time keeps retrying. + @Test(arguments: [ + "ssh: connect to host example.com port 22: Connection refused", + "kex_exchange_identification: Connection reset by peer", + "Connection closed by 10.0.0.5 port 22", + "no server running on /tmp/tmux-501/default", + "user@host: Permission denied (publickey).", + "", + ]) + func recoverableFailuresKeepRetrying(_ stderr: String) { + #expect(!RemoteTmuxSSHTransport.indicatesUnrecoverableTransportFailure(stderr)) + } + @Test func etCanTargetAServerWhoseTerminalIsNotOnThePath() { let profile = RemoteTmuxETTransportProfile( port: 2022, remoteTerminalPath: "/usr/local/bin/etterminal" From f961425e145bd6c85665ad4a8acd1e3b1b316da9 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 13:53:35 -0700 Subject: [PATCH 12/23] remote-tmux: a stream that never reached control mode is terminal, not retryable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit End-of-stream became "always reconnect" to stop discarding sessions that were still alive, which was right for the case it fixed and wrong for a transport that never started. Retrying there turned a precise error — "tmux control stream ended before attach" — into an opaque 60-second attach timeout, and that swallowed reason is what made a wedged etserver take most of a session to diagnose: every arm of the investigation saw a timeout instead of the cause. So the decision goes back into the pure type, with a parameter that actually decides. Reached control mode and then ended: something was there and may still be, so reconnect and let the reattach report whether the session is gone. Never reached it: the transport failed to start, there is no session behind it, and reporting beats retrying. The connection passes `enterReceived`, and the fuzz model now says out loud that its stream has connected by the point it tests an exit. Verified end to end against a real etserver on both sides of the change: the failure now names itself, and a healthy attach still parses the handshake and mirrors windows. --- Sources/RemoteTmuxControlConnection.swift | 6 +++-- Sources/RemoteTmuxTransportRegistry.swift | 17 +++++++++---- .../RemoteTmuxProxyTransportRetryTests.swift | 24 ++++++++++++++++--- 3 files changed, 37 insertions(+), 10 deletions(-) diff --git a/Sources/RemoteTmuxControlConnection.swift b/Sources/RemoteTmuxControlConnection.swift index 0df4e94f260a..948d86c99213 100644 --- a/Sources/RemoteTmuxControlConnection.swift +++ b/Sources/RemoteTmuxControlConnection.swift @@ -751,12 +751,14 @@ final class RemoteTmuxControlConnection { observers.notifyExit() return } - switch RemoteTmuxStreamEndDisposition.forStreamEnd() { + switch RemoteTmuxStreamEndDisposition.forStreamEnd(hasReachedControlMode: enterReceived) { case .reconnect: // Keep the mirror frozen and reconnect. beginReconnecting() case .sessionOver: - record("stream-end-session-over") + // Either the session ended, or the transport never started — both are terminal, and + // both must report rather than retry. + record(enterReceived ? "stream-end-session-over" : "stream-end-before-connect") connectionState = .ended cancelScheduledWork() teardownProcessHandles() diff --git a/Sources/RemoteTmuxTransportRegistry.swift b/Sources/RemoteTmuxTransportRegistry.swift index 2d47a28d4245..5bef6e688579 100644 --- a/Sources/RemoteTmuxTransportRegistry.swift +++ b/Sources/RemoteTmuxTransportRegistry.swift @@ -116,12 +116,19 @@ enum RemoteTmuxStreamEndDisposition: Sendable, Equatable { /// The transport owned recovery, so its exit is terminal. case sessionOver - /// What EOF on the control stream means, for every transport: reconnect and find out. + /// What EOF on the control stream means. /// - /// Deliberately takes no argument. It used to branch on who owns reconnection, and that branch - /// was wrong (see this type's documentation) — a parameter that no longer decides anything - /// would just invite the same inference back. - static func forStreamEnd() -> RemoteTmuxStreamEndDisposition { .reconnect } + /// It used to branch on who owns reconnection, and that branch was wrong (see this type's + /// documentation). The distinction that does hold is whether the stream ever worked: + /// + /// - reached control mode, then ended: something was there and may still be. Reconnect, and let + /// the reattach report whether the session is gone. + /// - never reached control mode: the transport failed to *start*. There is no session behind it + /// to preserve, and retrying only hides the reason — measured, it turned "tmux control stream + /// ended before attach" into an opaque 60-second attach timeout. + static func forStreamEnd(hasReachedControlMode: Bool) -> RemoteTmuxStreamEndDisposition { + hasReachedControlMode ? .reconnect : .sessionOver + } } /// A command to run before opening a connection to a host. diff --git a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift index 70abf4adeea3..b8d6918471d2 100644 --- a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift +++ b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift @@ -193,8 +193,8 @@ import Testing /// reasoning that such a transport does not end for a network drop. Measured against /// et 6.2.11+7: restarting only `etserver` ends the stream while `tmux has-session` still /// succeeds, so that reasoning discarded live, reattachable sessions. - @Test func endOfStreamAlwaysMeansReconnectAndLetTheReattachDecide() { - #expect(RemoteTmuxStreamEndDisposition.forStreamEnd() == .reconnect) + @Test func endOfStreamOnAnEstablishedStreamMeansReconnectAndLetTheReattachDecide() { + #expect(RemoteTmuxStreamEndDisposition.forStreamEnd(hasReachedControlMode: true) == .reconnect) } // MARK: - Seam 3: the pre-connect hook @@ -453,6 +453,22 @@ import Testing #expect(connection.snapshot().recentEvents.contains("liveness-stalled")) } + /// A stream that never reached control mode is a failed start, not a lost session. + /// + /// Reconnecting there is what made a real error — "tmux control stream ended before attach" — + /// surface as an opaque 60-second attach timeout, which cost most of a debugging session. + /// Reconnecting is right only once a stream has actually worked. + @Test func endOfStreamBeforeControlModeIsTerminalRatherThanRetried() { + #expect( + RemoteTmuxStreamEndDisposition.forStreamEnd(hasReachedControlMode: false) == .sessionOver, + "a transport that never started has nothing to reconnect to" + ) + #expect( + RemoteTmuxStreamEndDisposition.forStreamEnd(hasReachedControlMode: true) == .reconnect, + "an established stream may have a session still there — reattach decides" + ) + } + /// ssh must be untouched by all of this: it gets an EOF, `handleStreamEnd` already recovers, /// and probing an idle ssh stream would add traffic and a new way to fail. @MainActor @Test func anSSHTransportIsNeverProbedForStalls() { @@ -863,7 +879,9 @@ private struct SplitMix64 { // not exit for a mere network drop — measured against et 6.2.11+7, restarting only // `etserver` exits while `tmux has-session` still succeeds, so ending here threw // away a live session. Reconnect, and let the reattach report whether it is gone. - let disposition = RemoteTmuxStreamEndDisposition.forStreamEnd() + // The model's stream has reached control mode by this point, which is the case + // where an exit may still have a live session behind it. + let disposition = RemoteTmuxStreamEndDisposition.forStreamEnd(hasReachedControlMode: true) #expect(disposition == .reconnect, "an exit did not lead to a reattach — \(context)") model.spawnCount += 1 From 48dc4c49f2ed0188c559d64f55ca86f52392c828 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 16:49:22 -0700 Subject: [PATCH 13/23] remote-tmux: fold review findings on the transport seam Four findings from review, each verified against this branch rather than taken on faith: - The MAX_CANON test could not fail. `try? #require(...)` turned the throw into nil, so byteCount was 0 and 0 < 1024 passed even if the profile stopped emitting an `exec ` argument at all. Use `try #require`. - The conformance script called bare `timeout`, which stock macOS does not have. Every et_run then exited 127, and the ">MAX_CANON is not delivered" check read that as the claim holding, so the file whose thesis is "validate the claim" was itself passing for the wrong reason. Resolve the deadline command up front and refuse to run without a usable one. - `et` gets `--` before the destination, for the same reason ssh's argv does: a destination beginning with `-` has to be a host, never an option. Measured against et 6.2.11+7, which reports `-weirdhost` as unreachable with the guard and swallows it as options without. - The polling lint matched `DispatchQueue.<...>asyncAfter`, so it missed `DispatchQueue.global(qos:).asyncAfter` and any stored-queue receiver. Matching `.asyncAfter(` closes that gap; the hit set on this branch is unchanged, so nothing needed baselining. --- Sources/RemoteTmuxTransportRegistry.swift | 8 ++++++-- cmuxTests/RemoteTmuxProxyTransportRetryTests.swift | 6 +++--- scripts/lint-remote-tmux-no-polling.sh | 5 ++++- scripts/remote-tmux-et-conformance.sh | 13 +++++++++++-- 4 files changed, 24 insertions(+), 8 deletions(-) diff --git a/Sources/RemoteTmuxTransportRegistry.swift b/Sources/RemoteTmuxTransportRegistry.swift index 5bef6e688579..a9572f2192b4 100644 --- a/Sources/RemoteTmuxTransportRegistry.swift +++ b/Sources/RemoteTmuxTransportRegistry.swift @@ -387,8 +387,12 @@ struct RemoteTmuxETTransportProfile: RemoteTmuxTransportProfile { if let identityFile = host.identityFile { argv += ["--ssh-option", "IdentityFile=\(identityFile)"] } - // `exec` so the login shell does not linger as a parent of tmux. - argv += ["-c", "exec \(remote)", host.destination] + // `exec` so the login shell does not linger as a parent of tmux. `--` ends et's option + // parsing for the same reason ssh's argv does: a destination beginning with `-` has to be a + // host, never an option. Measured against et 6.2.11+7 - with the guard, `-weirdhost` is + // reported as an unreachable host; without it, et swallows it and exits "Missing host to + // connect to". + argv += ["-c", "exec \(remote)", "--", host.destination] return argv } diff --git a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift index b8d6918471d2..5484e75a8d8e 100644 --- a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift +++ b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift @@ -340,7 +340,7 @@ import Testing /// /// Long session names are the realistic way to cross the line, so the bound is checked /// against one rather than only the short names the other tests use. - @Test func etCommandFitsWhatALoginShellCanRead() { + @Test func etCommandFitsWhatALoginShellCanRead() throws { let maxCanon = 1024 for session in ["s", "work session", String(repeating: "session-", count: 24)] { let argv = RemoteTmuxETTransportProfile(port: 2039).controlStreamArgv( @@ -348,8 +348,8 @@ import Testing sessionName: session, createIfMissing: false ) - let command = try? #require(argv.first(where: { $0.hasPrefix("exec ") })) - let byteCount = (command ?? "").utf8.count + let command = try #require(argv.first(where: { $0.hasPrefix("exec ") })) + let byteCount = command.utf8.count #expect( byteCount < maxCanon, Comment( diff --git a/scripts/lint-remote-tmux-no-polling.sh b/scripts/lint-remote-tmux-no-polling.sh index 7d1361fb3915..c41abe576216 100755 --- a/scripts/lint-remote-tmux-no-polling.sh +++ b/scripts/lint-remote-tmux-no-polling.sh @@ -28,7 +28,10 @@ cd "$(dirname "$0")/.." || exit 1 SCOPE=(Sources/RemoteTmux*.swift Sources/RemoteTmuxController+*.swift) # Primitives that make a wait time-based rather than event-based. -PATTERN='Task\.sleep|Thread\.sleep|usleep\(|DispatchQueue\.[A-Za-z.]*asyncAfter|DispatchSourceTimer|Timer\.scheduledTimer|ContinuousClock\(\)\.sleep' +# `\.asyncAfter\(` rather than a `DispatchQueue.…` prefix: the receiver can be any expression +# (`DispatchQueue.global(qos: .background)`, a stored queue), and every one of them is a +# time-based wait. Matching the call, not the queue, closes that gap. +PATTERN='Task\.sleep|Thread\.sleep|usleep\(|\.asyncAfter\(|DispatchSourceTimer|Timer\.scheduledTimer|ContinuousClock\(\)\.sleep' # Waits that predate this guard, recorded so it blocks NEW ones without pretending the # existing ones are all fine. Several are worth revisiting — the sizing debounces in diff --git a/scripts/remote-tmux-et-conformance.sh b/scripts/remote-tmux-et-conformance.sh index eb9570046b33..40d6465a3fc7 100755 --- a/scripts/remote-tmux-et-conformance.sh +++ b/scripts/remote-tmux-et-conformance.sh @@ -56,6 +56,15 @@ for tool in ET_CLIENT ET_SERVER ET_TERMINAL; do fi done +# Resolve the deadline command before anything uses it. Stock macOS has no `timeout`, and a +# missing one makes every et_run exit 127 - which the ">MAX_CANON is not delivered" check reads +# as the claim holding. Fail loudly instead of passing for the wrong reason. +TIMEOUT_BIN="${CMUX_TIMEOUT_BIN:-$(command -v timeout || command -v gtimeout || true)}" +if [ -z "$TIMEOUT_BIN" ] || [ ! -x "$TIMEOUT_BIN" ]; then + echo "no usable timeout(1)/gtimeout(1) at '${TIMEOUT_BIN:-}'; brew install coreutils, or point CMUX_TIMEOUT_BIN at one" >&2 + exit 2 +fi + VERSION="$("$ET_CLIENT" --version 2>&1 | head -1)" echo "=== conformance against: $VERSION" echo " client=$ET_CLIENT server=$ET_SERVER terminal=$ET_TERMINAL" @@ -79,8 +88,8 @@ nc -z 127.0.0.1 "$PORT" 2>/dev/null || { echo "etserver did not start on $PORT" # path named rather than assumed. et_run() { local timeout_s="$1" command="$2" - timeout "$timeout_s" /usr/bin/script -q /dev/null \ - "$ET_CLIENT" -p "$PORT" --terminal-path "$ET_TERMINAL" -c "$command" "$HOST" 2>&1 + "$TIMEOUT_BIN" "$timeout_s" /usr/bin/script -q /dev/null \ + "$ET_CLIENT" -p "$PORT" --terminal-path "$ET_TERMINAL" -c "$command" -- "$HOST" 2>&1 } echo "--- claim: the remote command runs in a LOGIN shell, so it inherits the user's PATH" From 70a506de7285c45234bbcad867f96feced9ad36a Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 18:04:07 -0700 Subject: [PATCH 14/23] remote-tmux: wait on the server being gone, and say what the et host must be The teardown killed etserver and slept a second before checking that the tmux session outlived it. The claim only means something once the transport is actually down, so wait for that instead of guessing. etserver daemonizes itself, so it is not this script's child and `wait` cannot see it. Also record what CMUX_ET_HOST has to be. A reviewer read the default `cmux-ethost` as a stray hostname and suggested 127.0.0.1, but et bootstraps over ssh before its own protocol takes over, so the destination has to be something ssh can log into and the alias is what carries the user and host-key policy. The usage block now says that and shows the loopback alias. --- scripts/remote-tmux-et-conformance.sh | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/scripts/remote-tmux-et-conformance.sh b/scripts/remote-tmux-et-conformance.sh index 40d6465a3fc7..2d95344d29fe 100755 --- a/scripts/remote-tmux-et-conformance.sh +++ b/scripts/remote-tmux-et-conformance.sh @@ -29,6 +29,15 @@ # ET_CLIENT=/path/to/et ET_SERVER=/path/to/etserver \ # ET_TERMINAL=/path/to/etterminal scripts/remote-tmux-et-conformance.sh # +# The client connects to CMUX_ET_HOST (default `cmux-ethost`), which must be an ssh +# destination this machine can log into, because et bootstraps over ssh before its own +# protocol takes over. A loopback alias is enough: +# +# Host cmux-ethost +# HostName 127.0.0.1 +# +# Set CMUX_ET_HOST to point at a different one. +# # Exit code is the number of failed checks (0 = every belief holds). # ============================================================================ set -uo pipefail @@ -138,9 +147,17 @@ fi echo "--- claim: end-of-stream does NOT mean the remote session is gone" # cmux used to treat an ET exit as the session ending, and removed the mirror. Restarting only # etserver falsifies that: the stream ends, the session lives. -kill "$(cat "$PIDFILE" 2>/dev/null)" 2>/dev/null +SERVER_PID="$(cat "$PIDFILE" 2>/dev/null)" +kill "$SERVER_PID" 2>/dev/null rm -f "$PIDFILE" -sleep 1 +# Wait for the server to be gone rather than guessing at a second: the claim under test is +# that the session outlives the transport, so the transport has to be down before it is +# checked. etserver daemonizes itself, so it is not this script's child and `wait` cannot +# see it. +for _ in $(seq 1 40); do + kill -0 "$SERVER_PID" 2>/dev/null || break + sleep 0.25 +done if tmux has-session -t "$SESSION" 2>/dev/null; then pass "the session survives the transport dying, so EOF must lead to a reattach" else From 550c65a0219626b3b4f12b0670729a53d19177fe Mon Sep 17 00:00:00 2001 From: ejc3 Date: Tue, 21 Jul 2026 19:38:20 -0700 Subject: [PATCH 15/23] remote-tmux: ask the host before calling a silent stream wedged The stall monitor treated a probe still unanswered at the next tick as a wedged stream and respawned the transport. A real network interruption looks identical from the stream's side: the transport is reconnecting underneath and cannot answer a probe either, so an outage lasting longer than one interval terminated the process and discarded the session it was in the middle of resuming. An unanswered probe is now a suspicion, and the question that settles it is asked somewhere else. One-shot commands ride ssh's shared master even for an et connection, so tmux has-session reaches the host over a channel this stream's wedge cannot touch. That is the same asymmetry the seam doc already records measuring: restarting etserver closes the control stream while has-session keeps succeeding. A host that answers proves the stream is the broken part, which is the case to recover. A host that does not answer is the outage this transport exists to ride out, so the connection stays connected and asks again next tick. Deferral is bounded at four consecutive ticks. A host that is both unreachable and wedged would otherwise leave a frozen mirror that never retries, which is the failure the monitor was added to prevent. Reachability is injected the same way the transport profile is, so the tests decide the answer without a host or a spawned process. --- ...RemoteTmuxControlConnection+Commands.swift | 97 +++++++++++++++++-- Sources/RemoteTmuxControlConnection.swift | 61 +++++++++++- .../RemoteTmuxProxyTransportRetryTests.swift | 80 ++++++++++++--- docs/remote-tmux-transport-seam.md | 15 ++- 4 files changed, 228 insertions(+), 25 deletions(-) diff --git a/Sources/RemoteTmuxControlConnection+Commands.swift b/Sources/RemoteTmuxControlConnection+Commands.swift index e73d0bd92503..7536162a6ffd 100644 --- a/Sources/RemoteTmuxControlConnection+Commands.swift +++ b/Sources/RemoteTmuxControlConnection+Commands.swift @@ -119,27 +119,40 @@ extension RemoteTmuxControlConnection { /// without this check a wedged et connection stays `.connected` forever and the mirror /// freezes with no error and no retry. /// + /// A silent stream alone does not say which of those two it is. A real network interruption + /// silences the stream too, and the transport cannot answer a probe while it is reconnecting + /// underneath — so treating silence as the verdict kills the process and throws away the + /// session it was in the middle of recovering. The unanswered probe is therefore a suspicion, + /// and the question that settles it is asked somewhere else: ``sessionReachability`` runs a + /// one-shot over ssh's shared master, a channel this stream's wedge cannot affect. A host that + /// answers there proves the network is fine and the stream is the broken part, which is the + /// case to recover. A host that does not answer is an outage, and an outage is what this + /// transport exists to ride out — so stay `.connected` and ask again next tick, bounded by + /// ``maxConsecutiveLivenessDeferrals`` so a host that is both unreachable and wedged still + /// gets its reconnect. + /// /// Only reachable for `reconnectsInternally` transports: ssh gets its EOF and must keep its /// existing behavior exactly, including staying quiet on an idle stream. /// - /// - Parameter completion: `true` if the stream answered (or the check did not apply), and - /// `false` if it was found wedged and recovery was started. + /// - Parameter completion: `true` if the stream answered, the check did not apply, or the + /// verdict was deferred; `false` if the stream was found wedged and recovery was started. + /// A deferral reports `true` because `false` means "recovery has started" — callers act on + /// that edge, and a deferral deliberately starts nothing. func checkLivenessAndRecoverIfStalled(completion: ((Bool) -> Void)? = nil) { guard transportProfile.reconnectsInternally, connectionState == .connected, !exited else { completion?(true) return } let generation = processGeneration - // An unanswered previous probe IS the stall. ET can accept stdin while producing no - // control output, so a probe can be written and simply never answered — without this the - // probes accumulate, every one of them still pending, and the connection sits - // `.connected` forever. The deadline is the next tick rather than a second timer: probe N - // must be answered before probe N+1 is due, which is a generous bound on a local + // A previous probe left unanswered is the suspected stall. ET can accept stdin while + // producing no control output, so a probe can be written and simply never answered — + // without this the probes accumulate, every one of them still pending, and the connection + // sits `.connected` forever. The deadline is the next tick rather than a second timer: + // probe N must be answered before probe N+1 is due, which is a generous bound on a local // round-trip and needs no clock of its own. if livenessProbeOutstanding { record("liveness-unanswered") - recoverFromStalledTransport() - completion?(false) + resolveSuspectedStall(generation: generation, completion: completion) return } livenessProbeOutstanding = true @@ -153,6 +166,8 @@ extension RemoteTmuxControlConnection { return } if answered { + // The stream is carrying the protocol, so any run of deferred ticks is over. + self.livenessDeferralCount = 0 completion?(true) } else { self.recoverFromStalledTransport() @@ -166,6 +181,70 @@ extension RemoteTmuxControlConnection { } } + /// Asks out of band whether the far end is reachable, then either recovers the stream or + /// defers the verdict to the next tick. See ``checkLivenessAndRecoverIfStalled(completion:)`` + /// for why the silent stream cannot answer this itself. + private func resolveSuspectedStall(generation: UInt64, completion: ((Bool) -> Void)?) { + // One query at a time. The one-shot bounds itself with ssh's `ConnectTimeout` and + // `ServerAlive*` rather than a deadline of its own, so on a dead network it can outlast a + // tick; a second query would ask about the same outage and could recover twice for one + // stall. Report `true` — this tick found nothing wedged, and the query already running is + // the one that decides. + guard !livenessReachabilityQueryInFlight else { + completion?(true) + return + } + livenessReachabilityQueryInFlight = true + // Read the session name now: a `rename-session` during the query would otherwise change + // which session the answer is about. + let reachability = sessionReachability + let host = self.host + let sessionName = self.sessionName + Task { @MainActor [weak self] in + let reachable = await reachability(host, sessionName) + guard let self else { return } + self.livenessReachabilityQueryInFlight = false + // A respawn overtook the query: the stream this answer describes is gone, so acting on + // it would recover a stream that was already replaced. + guard generation == self.processGeneration, self.connectionState == .connected else { + completion?(true) + return + } + // The probe came back while the query was out. The stream is answering after all, so + // there is nothing to recover regardless of what the host said. + guard self.livenessProbeOutstanding else { + completion?(true) + return + } + if reachable { + self.recoverFromStalledTransport() + completion?(false) + return + } + self.livenessDeferralCount += 1 + if self.livenessDeferralCount > Self.maxConsecutiveLivenessDeferrals { + // Long enough. Either the outage is not what is keeping the stream quiet, or it + // has lasted past anything this transport is going to recover from on its own. + self.record("liveness-deferral-exhausted") + self.recoverFromStalledTransport() + completion?(false) + return + } + self.record("liveness-deferred-unreachable") + completion?(true) + } + } + + /// Clears the probe bookkeeping so a fresh stream is judged on its own evidence. + /// + /// Leaves ``livenessReachabilityQueryInFlight`` alone deliberately: a query that is still + /// running clears that itself when it lands, and clearing it here would let a second query + /// start while the first is outstanding. + func resetLivenessProbeState() { + livenessProbeOutstanding = false + livenessDeferralCount = 0 + } + /// Replaces a transport that is alive but no longer carrying the protocol. /// /// Recovery is a respawn rather than an end: the remote session is very likely still there — diff --git a/Sources/RemoteTmuxControlConnection.swift b/Sources/RemoteTmuxControlConnection.swift index 948d86c99213..fb6b00db00a9 100644 --- a/Sources/RemoteTmuxControlConnection.swift +++ b/Sources/RemoteTmuxControlConnection.swift @@ -162,12 +162,30 @@ final class RemoteTmuxControlConnection { /// ``checkLivenessAndRecoverIfStalled(completion:)``). Nil for ssh, which gets an EOF instead. private var livenessTask: Task? /// Whether a liveness probe is still waiting for its answer. The next probe's due time is the - /// previous one's deadline, so this is what turns "no answer" into a detected stall. + /// previous one's deadline, so this is what turns "no answer" into a suspected stall. var livenessProbeOutstanding = false + /// Whether an out-of-band reachability query is still running. One at a time: the query rides + /// ssh, which bounds itself with `ConnectTimeout`/`ServerAlive*` rather than a deadline of its + /// own, so on a dead network it can outlast a tick. A second query would answer about the same + /// outage and could recover twice for one stall, so a tick that lands on top of one in flight + /// does nothing and lets the query it is waiting on decide. + var livenessReachabilityQueryInFlight = false + /// Consecutive ticks that found the stream silent AND the host unreachable. Reset when a probe + /// is answered and when the connection reconnects, so only an unbroken run counts. + var livenessDeferralCount = 0 /// How often to ask a self-reconnecting transport whether it is still carrying the protocol. /// Long enough that an ordinary reconnect finishes untouched, short enough that a wedged /// mirror is not left silently frozen. static var livenessProbeIntervalSeconds: UInt64 = 30 + /// How many consecutive unreachable ticks may be deferred before the stream is recovered + /// anyway. At the 30-second interval that is about two minutes of outage. + /// + /// The cap is what keeps deferral honest. A transport that is mid-reconnect deserves to be + /// left alone, but "unreachable" cannot be trusted forever: if the host is both unreachable + /// and the stream is wedged, deferring without a limit leaves a frozen mirror that never + /// retries — the exact failure this monitor was added to prevent. Recovering after the cap + /// costs a reconnect that the backoff would have performed anyway. + static let maxConsecutiveLivenessDeferrals = 4 /// Number of reconnect attempts since the last successful connect, driving the /// capped exponential backoff. Reset to 0 on a successful connect. private var reconnectAttemptCount = 0 @@ -346,15 +364,44 @@ final class RemoteTmuxControlConnection { /// overrides it (which tests do, to assert argv without spawning anything). let transportProfile: RemoteTmuxTransportProfile + /// Asks whether the session is still reachable, on a channel that does not run through this + /// connection's control stream — the one question that separates a wedged stream from a + /// transport that is busy reconnecting underneath. Supplied by the caller the same way + /// ``transportProfile`` is, defaulting to ``oneShotSessionReachability`` (which tests replace, + /// to decide the answer without a host). + let sessionReachability: @Sendable (RemoteTmuxHost, String) async -> Bool + + /// The production reachability check: one `tmux has-session` over ssh's shared control master. + /// + /// Independent of the wedged stream by construction. Even for an et connection, one-shot + /// commands ride ssh (see ``RemoteTmuxETTransportProfile/oneShotArgv(host:remoteCommand:)``), + /// so this asks over a different transport entirely — and against et 6.2.11+7 that difference + /// is measurable: restarting `etserver` closes the control stream while `has-session` keeps + /// succeeding. The master is already open and single-flighted, so the question costs a + /// round-trip and no authentication. + /// + /// Anything other than a clean exit counts as unreachable, including a session tmux says is + /// gone. That is deliberately coarse: a gone session normally arrives as `%exit` on the + /// control stream, and if it somehow does not, the deferral cap recovers within about two + /// minutes and the reattach classifies the end the way it always does. + static let oneShotSessionReachability: @Sendable (RemoteTmuxHost, String) async -> Bool = { + host, sessionName in + let transport = RemoteTmuxSSHTransport(host: host) + let result = try? await transport.runTmux(["has-session", "-t", sessionName]) + return result?.succeeded == true + } + init( host: RemoteTmuxHost, sessionName: String, createIfMissing: Bool = false, pendingPaneSeedByteLimit: Int = RemoteTmuxControlConnection.maximumPendingPaneSeedBytes, - transportProfile: RemoteTmuxTransportProfile? = nil + transportProfile: RemoteTmuxTransportProfile? = nil, + sessionReachability: (@Sendable (RemoteTmuxHost, String) async -> Bool)? = nil ) { self.transportProfile = transportProfile ?? host.transport.profile(port: host.transportPort, terminalPath: host.transportTerminalPath) + self.sessionReachability = sessionReachability ?? Self.oneShotSessionReachability self.host = host self.sessionName = sessionName self.createIfMissing = createIfMissing @@ -583,7 +630,7 @@ final class RemoteTmuxControlConnection { reconnectTask = nil livenessTask?.cancel() livenessTask = nil - livenessProbeOutstanding = false + resetLivenessProbeState() resetWindowListRequestCoalescing() cancelSizingFollowUps() pendingPostAttachAction = nil @@ -800,6 +847,11 @@ final class RemoteTmuxControlConnection { /// Starts the stall monitor for a transport that owns its own reconnection. /// + /// Each tick asks the stream a question and reads the previous tick's answer; a probe still + /// unanswered when the next one is due makes the stream a suspect, not a casualty. What + /// separates a wedged stream from one whose transport is busy reconnecting is asked out of + /// band — see ``checkLivenessAndRecoverIfStalled(completion:)``. + /// /// ssh is deliberately excluded: its stream ends on transport loss, `handleStreamEnd` already /// recovers from that, and probing an idle ssh stream would add traffic and a failure mode /// where today there is none. @@ -824,6 +876,9 @@ final class RemoteTmuxControlConnection { // The stream is dead: a close decision awaiting an activity query must // not hang for the whole backoff window — fail it onto the cache now. failPendingCommandTransactions() + // The reconnect this starts is the recovery the deferral was waiting for, so the run of + // deferred ticks ends here rather than carrying into the next stream's accounting. + resetLivenessProbeState() resetWindowListRequestCoalescing() cancelSizingFollowUps() // Subscriptions belong to the dying client, so forget them HERE, not in diff --git a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift index 5484e75a8d8e..4371b090fcb5 100644 --- a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift +++ b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift @@ -433,24 +433,82 @@ import Testing #expect(connection.snapshot().recentEvents.contains("liveness-stalled")) } - /// A probe that is written but never answered is the stall this monitor exists for: ET can - /// accept stdin while producing no control output. Before the deadline, probes accumulated and - /// the connection stayed `.connected` forever — the monitor could not detect the very case it - /// was added for. - @MainActor @Test func anUnansweredProbeIsTreatedAsAStall() { - let connection = RemoteTmuxControlConnection( - host: RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2039), - sessionName: "work" - ) + /// A probe that is written but never answered, on a host that still answers out of band, is + /// the stall this monitor exists for: ET can accept stdin while producing no control output. + /// Before the deadline, probes accumulated and the connection stayed `.connected` forever — + /// the monitor could not detect the very case it was added for. + /// + /// The reachable answer is what makes this a stall rather than an outage: the host is fine, so + /// the stream is the broken part. + @MainActor @Test func anUnansweredProbeOnAReachableHostIsTreatedAsAStall() async { + let connection = Self.etConnection(reachable: true) connection.handle(.enter) // Stand in for a probe that was written and never came back. connection.livenessProbeOutstanding = true - var reported: Bool? - connection.checkLivenessAndRecoverIfStalled { reported = $0 } + let reported = await Self.tick(connection) #expect(reported == false) #expect(connection.snapshot().recentEvents.contains("liveness-unanswered")) #expect(connection.snapshot().recentEvents.contains("liveness-stalled")) + #expect(connection.connectionState == .reconnecting) + } + + /// An unanswered probe during a real outage must not cost the session. + /// + /// The transport reconnects underneath, and while it does it cannot answer a probe either — so + /// silence alone cannot mean "wedged". Recovering here terminates the transport process and + /// discards the session it was in the middle of resuming. The host is asked out of band, over + /// a channel the wedge cannot reach, and an unreachable host means wait. + @MainActor @Test func anOutageDefersTheVerdictInsteadOfDiscardingTheSession() async { + let connection = Self.etConnection(reachable: false) + connection.handle(.enter) + connection.livenessProbeOutstanding = true + + let reported = await Self.tick(connection) + #expect(reported == true, "nothing was recovered, so there is no recovery edge to report") + #expect(connection.connectionState == .connected, "an outage must not end the session") + #expect(!connection.snapshot().recentEvents.contains("liveness-stalled")) + #expect(connection.snapshot().recentEvents.contains("liveness-deferred-unreachable")) + } + + /// Deferral is bounded. A host that stays unreachable while the stream stays silent is + /// eventually recovered anyway: if the outage is not what silenced the stream, waiting on it + /// forever leaves a frozen mirror with no retry — the failure this monitor was added for. + @MainActor @Test func aDeferralRunEndsInRecoveryOnceTheCapIsPassed() async { + let connection = Self.etConnection(reachable: false) + connection.handle(.enter) + connection.livenessProbeOutstanding = true + + let cap = RemoteTmuxControlConnection.maxConsecutiveLivenessDeferrals + for _ in 0.. RemoteTmuxControlConnection { + RemoteTmuxControlConnection( + host: RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2039), + sessionName: "work", + sessionReachability: { _, _ in reachable } + ) + } + + /// Runs one monitor tick and waits for its verdict. The verdict can now cross an `await` (the + /// out-of-band question), so it is read from the completion rather than after the call. + @MainActor private static func tick(_ connection: RemoteTmuxControlConnection) async -> Bool { + await withCheckedContinuation { continuation in + connection.checkLivenessAndRecoverIfStalled { continuation.resume(returning: $0) } + } } /// A stream that never reached control mode is a failed start, not a lost session. diff --git a/docs/remote-tmux-transport-seam.md b/docs/remote-tmux-transport-seam.md index 46e223f60a64..9296ba8a1bbf 100644 --- a/docs/remote-tmux-transport-seam.md +++ b/docs/remote-tmux-transport-seam.md @@ -106,8 +106,19 @@ transport that reconnects internally needs a liveness check instead: cmux already has the round-trip primitive: a bounded `display-message -p ok` query, used as `awaitCommandBarrier` in `RemoteTmuxViewConnection`. Reuse it rather than inventing a heartbeat, and give the probe a deadline: measured against 6.2.11+7, et can accept stdin while -producing no control output, so an unanswered probe is the stall. The next probe's due time is -that deadline, which needs no second clock. +producing no control output, so an unanswered probe is the suspicion. The next probe's due time +is that deadline, which needs no second clock. + +An unanswered probe cannot be the verdict, though, because a real network interruption looks +identical from the stream's side: the transport is reconnecting underneath and cannot answer +either, and recovering there kills the process and throws away the session it was resuming. +What tells the two apart is a question asked somewhere else. One-shot commands ride ssh's +shared master even for an et connection, so `tmux has-session` reaches the host over a channel +this stream's wedge cannot touch — the same asymmetry the `etserver` restart above measures. A +host that answers proves the stream is the broken part, and that is the case to recover. A host +that does not answer is an outage, so stay connected and ask again next tick, capped at four +consecutive deferrals (about two minutes) so a host that is both unreachable and wedged still +gets its reconnect. A transport exit does **not** mean the session is over, tempting as the symmetry is. Restarting only `etserver` ends the stream while `tmux has-session` still succeeds, so acting on it discarded From 8d4793b19cc75e2674b0da9f3fcfefc4ac70ddc2 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Fri, 24 Jul 2026 04:20:34 -0700 Subject: [PATCH 16/23] remote-tmux: stop the et harness advertising a tmux tmpdir it never uses The script created $DIR/tmux and printed it as "tmux tmpdir:", but TMUX_TMPDIR is never set anywhere in the harness, so the session lives on the default server. Telling the reader otherwise is worse than saying nothing: the whole reason the session is on the default server is spelled out a few lines above (cmux runs the has-session check over ssh while the control stream rides et, so a private tmpdir would leave the ssh side unable to find the session), and the line contradicted it. The printout already reports the socket the session is actually on, which is the fact worth having. What protects a developer's own tmux is unchanged and is the guard below: the harness refuses to adopt a session it did not create. --- scripts/remote-tmux-et-host.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/scripts/remote-tmux-et-host.sh b/scripts/remote-tmux-et-host.sh index 40ae0f6ffb74..ae1d83f25ba2 100755 --- a/scripts/remote-tmux-et-host.sh +++ b/scripts/remote-tmux-et-host.sh @@ -25,7 +25,7 @@ if [ -L "$STATE_ROOT" ] || [ -L "$DIR" ]; then echo "refusing symlinked et state path" >&2; exit 1 fi umask 077 -mkdir -p "$DIR/logs" "$DIR/tmux" +mkdir -p "$DIR/logs" chmod 700 "$DIR" # etserver wants a pidfile it can write; /var/run needs root, so keep it local. @@ -83,7 +83,6 @@ cat <' $USER@127.0.0.1 INFO From e625c4ea5bbe69583d73ea2fe21e46ecc414f089 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Fri, 24 Jul 2026 23:07:44 -0700 Subject: [PATCH 17/23] remote-tmux: default the et conformance host to loopback Nothing in the repo maps the name cmux-ethost, so a fresh checkout of the conformance script pointed at a destination that did not resolve. Default CMUX_ET_HOST to 127.0.0.1, which is self-explanatory and matches the loopback setup the script assumes; override it when sshd lives elsewhere. --- scripts/remote-tmux-et-conformance.sh | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/scripts/remote-tmux-et-conformance.sh b/scripts/remote-tmux-et-conformance.sh index 2d95344d29fe..c0045aad29bd 100755 --- a/scripts/remote-tmux-et-conformance.sh +++ b/scripts/remote-tmux-et-conformance.sh @@ -29,14 +29,11 @@ # ET_CLIENT=/path/to/et ET_SERVER=/path/to/etserver \ # ET_TERMINAL=/path/to/etterminal scripts/remote-tmux-et-conformance.sh # -# The client connects to CMUX_ET_HOST (default `cmux-ethost`), which must be an ssh +# The client connects to CMUX_ET_HOST (default `127.0.0.1`), which must be an ssh # destination this machine can log into, because et bootstraps over ssh before its own -# protocol takes over. A loopback alias is enough: -# -# Host cmux-ethost -# HostName 127.0.0.1 -# -# Set CMUX_ET_HOST to point at a different one. +# protocol takes over. The loopback default works wherever this machine runs sshd and +# accepts key auth for the current user; point CMUX_ET_HOST at a different destination +# (an ssh_config alias, another host) when that is not the case. # # Exit code is the number of failed checks (0 = every belief holds). # ============================================================================ @@ -50,7 +47,7 @@ ET_SERVER="${ET_SERVER:-$(command -v etserver || true)}" # hardcoding one of these is itself one of the defects this file exists to prevent. ET_TERMINAL="${ET_TERMINAL:-$(command -v etterminal || true)}" PORT="${CMUX_ET_PORT:-2041}" -HOST="${CMUX_ET_HOST:-cmux-ethost}" +HOST="${CMUX_ET_HOST:-127.0.0.1}" SESSION="cmux-conformance-$$" FAILURES=0 From 4ab32b55ca8741b230671a8db6c46954a71ce940 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Fri, 7 Aug 2026 22:31:17 -0700 Subject: [PATCH 18/23] remote-tmux: document the ssh-tmux transport flags --transport and --transport-port have been parsed since this branch added them, but nothing said so: `cmux ssh-tmux --help` listed only --port, --identity and --no-focus, so the only way to find the flags was to read the argument parser. Add them to the usage line, the flag list, and an example, and note the two defaults that are easy to get wrong -- the port belongs to the transport, so it is sshd's for ssh and etserver's for et. The other 19 translations of the help string are marked needs_review, since the English they were translated from has changed. --- CLI/cmux.swift | 15 +++++++++---- Resources/Localizable.xcstrings | 40 ++++++++++++++++----------------- 2 files changed, 31 insertions(+), 24 deletions(-) diff --git a/CLI/cmux.swift b/CLI/cmux.swift index 82ecc5012e7f..5a3067fc3db0 100644 --- a/CLI/cmux.swift +++ b/CLI/cmux.swift @@ -19299,7 +19299,8 @@ struct CMUXCLI { return Self.moshTmuxCommandUsage case "ssh-tmux": let help = String(localized: "cli.help.ssh-tmux", defaultValue: """ - Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus] + Usage: cmux ssh-tmux [--port ] [--identity ] + [--transport ssh|et] [--transport-port ] [--no-focus] Mirror a remote host's tmux sessions into the current window's sidebar over SSH tmux control mode (tmux -CC). Each session becomes a workspace, each @@ -19313,13 +19314,19 @@ struct CMUXCLI { with no prompt. ~/.ssh/config aliases and their IdentityFile/ProxyJump/Port settings are honored. Flags: - --port SSH port - --identity SSH identity file path - --no-focus Do not select the mirror workspace or focus its window + --port SSH port + --identity SSH identity file path + --transport ssh (default) or et. et carries the control stream over + EternalTerminal, which reconnects on its own, so the + mirror survives a network change instead of respawning. + --transport-port Port the transport connects to: sshd's for ssh, etserver's + for et (default 22 for ssh, 2022 for et) + --no-focus Do not select the mirror workspace or focus its window Example: cmux ssh-tmux dev@my-host cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519 + cmux ssh-tmux dev@my-host --transport et --transport-port 8080 """) let newWindowHelp = String( localized: "cli.help.ssh-tmux.newWindow", diff --git a/Resources/Localizable.xcstrings b/Resources/Localizable.xcstrings index ef796cc708b0..db91e796584e 100644 --- a/Resources/Localizable.xcstrings +++ b/Resources/Localizable.xcstrings @@ -87285,121 +87285,121 @@ "localizations": { "ar": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nاعكس جلسات tmux للمضيف البعيد داخل نافذة cmux الحالية عبر وضع تحكم tmux بواسطة SSH. تصبح كل جلسة مساحة عمل، وكل نافذة علامة تبويب، وكل نافذة متعددة الأجزاء تقسيمًا أصليًا. يتطلب ذلك تفعيل الإصدار التجريبي \"Remote tmux\".\n\nعند الحاجة إلى مصادقة تفاعلية، يشغّل cmux أمر ssh في هذه الطرفية ثم يعيد المحاولة عبر الاتصال المشترك. تُحترم الأسماء المستعارة وإعدادات ~/.ssh/config.\n\nFlags:\n --port منفذ SSH\n --identity مسار ملف هوية SSH\n --no-focus عدم تحديد مساحة العمل المعكوسة أو تنشيط نافذتها\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "bs": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nPreslikava tmux sesije udaljenog hosta u trenutni cmux prozor putem SSH tmux kontrolnog režima. Svaka sesija postaje radni prostor, svaki prozor kartica, a prozor s više okana izvorna podjela. Potrebno je uključiti beta opciju \"Remote tmux\".\n\nAko je potrebna interaktivna autentifikacija, cmux pokreće ssh u ovom terminalu i zatim pokušava ponovo preko dijeljene veze. Poštuju se aliasi i postavke iz ~/.ssh/config.\n\nFlags:\n --port SSH port\n --identity putanja SSH identiteta\n --no-focus ne biraj preslikani radni prostor niti aktiviraj prozor\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "da": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nSpejl en fjernværts tmux-sessioner i det aktuelle cmux-vindue via SSH tmux-kontroltilstand. Hver session bliver et arbejdsområde, hvert vindue en fane og hvert vindue med flere ruder en integreret opdeling. Kræver at betaindstillingen \"Remote tmux\" er slået til.\n\nHvis interaktiv godkendelse er nødvendig, kører cmux ssh i denne terminal og prøver derefter igen via den delte forbindelse. Aliasser og indstillinger i ~/.ssh/config respekteres.\n\nFlags:\n --port SSH-port\n --identity sti til SSH-identitetsfil\n --no-focus vælg ikke det spejlede arbejdsområde eller aktivér vinduet\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "de": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nSpiegelt die tmux-Sitzungen eines entfernten Hosts per SSH-tmux-Steuermodus in das aktuelle cmux-Fenster. Jede Sitzung wird zu einem Arbeitsbereich, jedes Fenster zu einem Tab und jedes Fenster mit mehreren Bereichen zu einer nativen Teilung. Erfordert die aktivierte Betaoption \"Remote tmux\".\n\nWenn eine interaktive Authentifizierung nötig ist, führt cmux ssh in diesem Terminal aus und versucht es anschließend über die gemeinsame Verbindung erneut. Aliase und Einstellungen aus ~/.ssh/config werden berücksichtigt.\n\nFlags:\n --port SSH-Port\n --identity Pfad zur SSH-Identitätsdatei\n --no-focus Spiegel-Arbeitsbereich nicht auswählen und Fenster nicht aktivieren\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "en": { "stringUnit": { "state": "translated", - "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nMirror a remote host's tmux sessions into the current cmux window over SSH tmux control mode. Each session becomes a workspace, each window a tab, and each multi-pane window a native split. Requires the \"Remote tmux\" beta setting.\n\nIf interactive authentication is needed, cmux runs ssh in this terminal and then retries over the shared connection. ~/.ssh/config aliases and settings are honored.\n\nFlags:\n --port SSH port\n --identity SSH identity file path\n --no-focus Do not select the mirror workspace or activate its window\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" + "value": "Usage: cmux ssh-tmux [--port ] [--identity ]\n [--transport ssh|et] [--transport-port ] [--no-focus]\n\nMirror a remote host's tmux sessions into the current window's sidebar over\nSSH tmux control mode (tmux -CC). Each session becomes a workspace, each\nwindow a tab, and each multi-pane window a native split. Requires the\n\"Remote tmux\" beta setting.\n\nIf the host needs interactive authentication (password, host-key confirmation,\nMFA, or a security-key touch), cmux runs ssh inline in this terminal so you can\nauthenticate, then mirrors the sessions over the shared SSH connection. Hosts\nthat authenticate non-interactively (ssh-agent / key in ~/.ssh/config) mirror\nwith no prompt. ~/.ssh/config aliases and their IdentityFile/ProxyJump/Port settings are honored.\n\nFlags:\n --port SSH port\n --identity SSH identity file path\n --transport ssh (default) or et. et carries the control stream over\n EternalTerminal, which reconnects on its own, so the\n mirror survives a network change instead of respawning.\n --transport-port Port the transport connects to: sshd's for ssh, etserver's\n for et (default 22 for ssh, 2022 for et)\n --no-focus Do not select the mirror workspace or focus its window\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519\n cmux ssh-tmux dev@my-host --transport et --transport-port 8080" } }, "es": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nRefleja las sesiones tmux de un host remoto en la ventana actual de cmux mediante el modo de control tmux por SSH. Cada sesión se convierte en un espacio de trabajo, cada ventana en una pestaña y cada ventana con varios paneles en una división nativa. Requiere activar la función beta \"Remote tmux\".\n\nSi se necesita autenticación interactiva, cmux ejecuta ssh en este terminal y luego reintenta mediante la conexión compartida. Se respetan los alias y ajustes de ~/.ssh/config.\n\nFlags:\n --port puerto SSH\n --identity ruta del archivo de identidad SSH\n --no-focus no seleccionar el espacio reflejado ni activar su ventana\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "fr": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nReproduit les sessions tmux d’un hôte distant dans la fenêtre cmux actuelle via le mode de contrôle tmux sur SSH. Chaque session devient un espace de travail, chaque fenêtre un onglet et chaque fenêtre à plusieurs volets une division native. Nécessite l’option bêta \"Remote tmux\".\n\nSi une authentification interactive est nécessaire, cmux exécute ssh dans ce terminal puis réessaie via la connexion partagée. Les alias et réglages de ~/.ssh/config sont respectés.\n\nFlags:\n --port port SSH\n --identity chemin du fichier d’identité SSH\n --no-focus ne pas sélectionner l’espace miroir ni activer sa fenêtre\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "it": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nReplica le sessioni tmux di un host remoto nella finestra cmux corrente tramite la modalità di controllo tmux su SSH. Ogni sessione diventa uno spazio di lavoro, ogni finestra una scheda e ogni finestra con più riquadri una suddivisione nativa. Richiede l’opzione beta \"Remote tmux\".\n\nSe serve l’autenticazione interattiva, cmux esegue ssh in questo terminale e poi riprova tramite la connessione condivisa. Gli alias e le impostazioni di ~/.ssh/config vengono rispettati.\n\nFlags:\n --port porta SSH\n --identity percorso del file di identità SSH\n --no-focus non selezionare lo spazio replicato né attivarne la finestra\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "ja": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nSSH の tmux コントロールモードで、リモートホストの tmux セッションを現在の cmux ウィンドウにミラーリングします。各セッションはワークスペースに、各ウィンドウはタブに、複数ペインのウィンドウはネイティブな分割になります。「Remote tmux」ベータ設定を有効にする必要があります。\n\n対話型認証が必要な場合、cmux はこのターミナルで ssh を実行し、共有接続を使って再試行します。~/.ssh/config のエイリアスと設定も使用されます。\n\nFlags:\n --port SSH ポート\n --identity SSH 識別ファイルのパス\n --no-focus ミラーワークスペースを選択せず、ウィンドウもアクティブにしない\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "km": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nឆ្លុះសម័យ tmux របស់ម៉ាស៊ីនពីចម្ងាយចូលក្នុងបង្អួច cmux បច្ចុប្បន្ន តាមរយៈរបៀបបញ្ជា tmux លើ SSH។ សម័យនីមួយៗក្លាយជាកន្លែងធ្វើការ បង្អួចនីមួយៗក្លាយជាផ្ទាំង និងបង្អួចដែលមានផ្ទាំងច្រើនក្លាយជាការបំបែកដើម។ ត្រូវបើកមុខងារសាកល្បង \"Remote tmux\"។\n\nបើត្រូវការផ្ទៀងផ្ទាត់អន្តរកម្ម cmux នឹងដំណើរការ ssh ក្នុងស្ថានីយនេះ ហើយសាកល្បងម្តងទៀតតាមការតភ្ជាប់រួម។ ឈ្មោះកាត់ និងការកំណត់ក្នុង ~/.ssh/config ត្រូវបានគោរព។\n\nFlags:\n --port ច្រក SSH\n --identity ផ្លូវឯកសារអត្តសញ្ញាណ SSH\n --no-focus កុំជ្រើសកន្លែងធ្វើការឆ្លុះ ឬបើកបង្អួចរបស់វា\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "ko": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nSSH tmux 제어 모드로 원격 호스트의 tmux 세션을 현재 cmux 창에 미러링합니다. 각 세션은 작업 공간이 되고, 각 창은 탭이 되며, 여러 패널이 있는 창은 네이티브 분할이 됩니다. \"Remote tmux\" 베타 설정을 켜야 합니다.\n\n대화형 인증이 필요하면 cmux가 이 터미널에서 ssh를 실행한 뒤 공유 연결을 통해 다시 시도합니다. ~/.ssh/config의 별칭과 설정도 적용됩니다.\n\nFlags:\n --port SSH 포트\n --identity SSH ID 파일 경로\n --no-focus 미러 작업 공간을 선택하거나 창을 활성화하지 않음\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "nb": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nSpeil tmux-øktene på en ekstern vert i det gjeldende cmux-vinduet via tmux-kontrollmodus over SSH. Hver økt blir et arbeidsområde, hvert vindu en fane og hvert vindu med flere ruter en integrert deling. Krever at betaalternativet \"Remote tmux\" er aktivert.\n\nHvis interaktiv autentisering er nødvendig, kjører cmux ssh i denne terminalen og prøver deretter på nytt via den delte forbindelsen. Aliaser og innstillinger i ~/.ssh/config blir brukt.\n\nFlags:\n --port SSH-port\n --identity bane til SSH-identitetsfil\n --no-focus ikke velg det speilede arbeidsområdet eller aktiver vinduet\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "pl": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nOdzwierciedla sesje tmux zdalnego hosta w bieżącym oknie cmux przez tryb sterowania tmux po SSH. Każda sesja staje się obszarem roboczym, każde okno kartą, a okno z wieloma panelami natywnym podziałem. Wymaga włączenia funkcji beta \"Remote tmux\".\n\nJeśli potrzebne jest interaktywne uwierzytelnianie, cmux uruchamia ssh w tym terminalu, a następnie ponawia próbę przez współdzielone połączenie. Aliasy i ustawienia z ~/.ssh/config są respektowane.\n\nFlags:\n --port port SSH\n --identity ścieżka pliku tożsamości SSH\n --no-focus nie wybieraj obszaru lustrzanego ani nie aktywuj jego okna\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "pt-BR": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nEspelha as sessões tmux de um host remoto na janela atual do cmux pelo modo de controle tmux via SSH. Cada sessão vira um espaço de trabalho, cada janela uma aba e cada janela com vários painéis uma divisão nativa. Requer a opção beta \"Remote tmux\".\n\nSe for necessária autenticação interativa, o cmux executa ssh neste terminal e tenta novamente pela conexão compartilhada. Os aliases e ajustes de ~/.ssh/config são respeitados.\n\nFlags:\n --port porta SSH\n --identity caminho do arquivo de identidade SSH\n --no-focus não selecionar o espaço espelhado nem ativar sua janela\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "ru": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nОтображает сеансы tmux удалённого узла в текущем окне cmux через режим управления tmux по SSH. Каждый сеанс становится рабочим пространством, каждое окно — вкладкой, а окно с несколькими панелями — нативным разделением. Требуется включить бета-функцию \"Remote tmux\".\n\nЕсли нужна интерактивная аутентификация, cmux запускает ssh в этом терминале, а затем повторяет попытку через общее соединение. Псевдонимы и настройки из ~/.ssh/config учитываются.\n\nFlags:\n --port порт SSH\n --identity путь к файлу идентификации SSH\n --no-focus не выбирать зеркальное пространство и не активировать его окно\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "th": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nมิเรอร์เซสชัน tmux ของโฮสต์ระยะไกลมายังหน้าต่าง cmux ปัจจุบันผ่านโหมดควบคุม tmux บน SSH แต่ละเซสชันจะเป็นพื้นที่ทำงาน แต่ละหน้าต่างจะเป็นแท็บ และหน้าต่างหลายบานจะเป็นการแบ่งแบบเนทีฟ ต้องเปิดใช้รุ่นเบต้า \"Remote tmux\"\n\nหากต้องยืนยันตัวตนแบบโต้ตอบ cmux จะเรียก ssh ในเทอร์มินัลนี้แล้วลองใหม่ผ่านการเชื่อมต่อที่ใช้ร่วมกัน โดยใช้ชื่อแทนและการตั้งค่าจาก ~/.ssh/config\n\nFlags:\n --port พอร์ต SSH\n --identity พาธไฟล์ข้อมูลประจำตัว SSH\n --no-focus ไม่เลือกพื้นที่ทำงานที่มิเรอร์หรือเปิดใช้งานหน้าต่าง\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "tr": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nUzak bir ana makinenin tmux oturumlarını SSH tmux denetim modu üzerinden geçerli cmux penceresine yansıtır. Her oturum bir çalışma alanına, her pencere bir sekmeye ve çok bölmeli her pencere yerel bir bölmeye dönüşür. \"Remote tmux\" beta ayarının etkin olması gerekir.\n\nEtkileşimli kimlik doğrulama gerekirse cmux bu terminalde ssh çalıştırır ve ardından paylaşılan bağlantı üzerinden yeniden dener. ~/.ssh/config içindeki takma adlar ve ayarlar kullanılır.\n\nFlags:\n --port SSH bağlantı noktası\n --identity SSH kimlik dosyası yolu\n --no-focus yansıtılan çalışma alanını seçme veya penceresini etkinleştirme\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "uk": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nВіддзеркалює сеанси tmux віддаленого вузла в поточному вікні cmux через режим керування tmux по SSH. Кожен сеанс стає робочим простором, кожне вікно — вкладкою, а вікно з кількома панелями — нативним поділом. Потрібно ввімкнути бета-функцію \"Remote tmux\".\n\nЯкщо потрібна інтерактивна автентифікація, cmux запускає ssh у цьому терміналі, а потім повторює спробу через спільне з’єднання. Псевдоніми й налаштування з ~/.ssh/config враховуються.\n\nFlags:\n --port порт SSH\n --identity шлях до файлу ідентифікації SSH\n --no-focus не вибирати віддзеркалений простір і не активувати його вікно\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "zh-Hans": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\n通过 SSH tmux 控制模式，将远程主机的 tmux 会话镜像到当前 cmux 窗口中。每个会话成为一个工作区，每个窗口成为一个标签页，多窗格窗口则成为原生分栏。需要启用“Remote tmux”测试功能。\n\n如果需要交互式身份验证，cmux 会在此终端中运行 ssh，然后通过共享连接重试。系统会采用 ~/.ssh/config 中的别名和设置。\n\nFlags:\n --port SSH 端口\n --identity SSH 身份文件路径\n --no-focus 不选择镜像工作区，也不激活其窗口\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } }, "zh-Hant": { "stringUnit": { - "state": "translated", + "state": "needs_review", "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\n透過 SSH tmux 控制模式，將遠端主機的 tmux 工作階段鏡像到目前的 cmux 視窗。每個工作階段會成為工作區，每個視窗會成為標籤頁，多窗格視窗則會成為原生分割。需要啟用「Remote tmux」測試功能。\n\n若需要互動式驗證，cmux 會在此終端機中執行 ssh，然後透過共用連線重試。系統會採用 ~/.ssh/config 中的別名與設定。\n\nFlags:\n --port SSH 連接埠\n --identity SSH 身分檔案路徑\n --no-focus 不選取鏡像工作區，也不啟用其視窗\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" } } From 965fd646543cdceac33dfa4933555a5ab5d9f76b Mon Sep 17 00:00:00 2001 From: ejc3 Date: Mon, 31 Aug 2026 09:33:58 -0700 Subject: [PATCH 19/23] remote-tmux: stop respawning a transport that reconnects on its own MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A mirror over EternalTerminal came back dead after a night away, with two orphaned client trees behind it. cmux had probed the stream every 30 seconds and respawned it when a probe went unanswered, on the theory that such a transport can be alive but wedged. An et client riding out a network change is quiet for longer than that, so the probe killed the process that was about to recover silently, and the replacement had to bootstrap a new session over ssh — which on a host with a second factor cannot happen unattended. The detector turned a recoverable pause into an unrecoverable one. The client's own exit is the trigger now, the same end-of-stream signal ssh has always used, and it is better informed than anything this side can infer: et exchanges keepalives with its server and gives up when they stop. That leaves one wedge this no longer covers — a remote etterminal that dies while the client keeps talking to etserver — which is a real trade for never destroying a healthy session, and wants a visible stale state rather than a respawn. Teardown also has to reach the whole transport. A pty allocator execs a broker that execs the client, and ^D% ]2;ejc3@ejc3-mac:~/src/cmux-rebase-fleet]1;..-rebase-fleet]7;file://ejc3-mac/Users/ejc3/src/cmux-rebase-fleet\ ➜ cmux-rebase-fleet [?1h=[?2004h[?2004l Script started, output file is typescript Script done, output file is typescript puts that payload in a process group of its own, so neither terminating the allocator nor signalling its group reaches the client (measured: different pgids, payload survived the group kill). The tree is walked instead, children first, SIGTERM then SIGKILL for anything still alive, with each group leader taking its group along. --- ...RemoteTmuxControlConnection+Commands.swift | 158 ++--------------- Sources/RemoteTmuxControlConnection.swift | 160 ++++++++---------- .../RemoteTmuxProxyTransportRetryTests.swift | 156 ++++++----------- 3 files changed, 140 insertions(+), 334 deletions(-) diff --git a/Sources/RemoteTmuxControlConnection+Commands.swift b/Sources/RemoteTmuxControlConnection+Commands.swift index 7536162a6ffd..cc25f7fd9d01 100644 --- a/Sources/RemoteTmuxControlConnection+Commands.swift +++ b/Sources/RemoteTmuxControlConnection+Commands.swift @@ -110,151 +110,27 @@ extension RemoteTmuxControlConnection { return sendTracked("display-message -p cmux-liveness", completion: completion) } - /// Checks a stalled-but-alive control stream and recovers it. + /// Whether the stream is answering, asked once on demand. /// - /// A transport that reconnects internally never delivers the EOF that drives ssh recovery: - /// during a network change its process stays up and the stream simply pauses. That is the - /// behavior worth having, but it means a transport that is alive and *not* recovering looks - /// exactly like one that is idle. Nothing else in the lifecycle can tell those apart, so - /// without this check a wedged et connection stays `.connected` forever and the mirror - /// freezes with no error and no retry. + /// There is deliberately no periodic version of this and no recovery attached to it. cmux + /// used to probe a self-reconnecting transport every 30 seconds and respawn it when a probe + /// went unanswered, on the theory that such a transport can be alive but wedged. Measured + /// against et on a host that requires a second factor, the cure was far worse: an et client + /// riding out a network change is quiet for longer than a tick, so the probe killed the one + /// process that could have recovered silently, and its replacement had to bootstrap a new + /// session over ssh — which needs an interactive second factor nobody is there to give at + /// 03:00. The mirror came back dead with two orphaned clients behind it. /// - /// A silent stream alone does not say which of those two it is. A real network interruption - /// silences the stream too, and the transport cannot answer a probe while it is reconnecting - /// underneath — so treating silence as the verdict kills the process and throws away the - /// session it was in the middle of recovering. The unanswered probe is therefore a suspicion, - /// and the question that settles it is asked somewhere else: ``sessionReachability`` runs a - /// one-shot over ssh's shared master, a channel this stream's wedge cannot affect. A host that - /// answers there proves the network is fine and the stream is the broken part, which is the - /// case to recover. A host that does not answer is an outage, and an outage is what this - /// transport exists to ride out — so stay `.connected` and ask again next tick, bounded by - /// ``maxConsecutiveLivenessDeferrals`` so a host that is both unreachable and wedged still - /// gets its reconnect. - /// - /// Only reachable for `reconnectsInternally` transports: ssh gets its EOF and must keep its - /// existing behavior exactly, including staying quiet on an idle stream. - /// - /// - Parameter completion: `true` if the stream answered, the check did not apply, or the - /// verdict was deferred; `false` if the stream was found wedged and recovery was started. - /// A deferral reports `true` because `false` means "recovery has started" — callers act on - /// that edge, and a deferral deliberately starts nothing. - func checkLivenessAndRecoverIfStalled(completion: ((Bool) -> Void)? = nil) { - guard transportProfile.reconnectsInternally, connectionState == .connected, !exited else { - completion?(true) - return - } - let generation = processGeneration - // A previous probe left unanswered is the suspected stall. ET can accept stdin while - // producing no control output, so a probe can be written and simply never answered — - // without this the probes accumulate, every one of them still pending, and the connection - // sits `.connected` forever. The deadline is the next tick rather than a second timer: - // probe N must be answered before probe N+1 is due, which is a generous bound on a local - // round-trip and needs no clock of its own. - if livenessProbeOutstanding { - record("liveness-unanswered") - resolveSuspectedStall(generation: generation, completion: completion) + /// et answers this itself: its client exchanges keepalives every few seconds and exits when + /// they stop, so a quiet client is idle or reconnecting, never wedged, and its exit already + /// reaches ``handleStreamEnd(processGeneration:)`` like any other end of stream. Trusting + /// that is both simpler and strictly better informed than guessing from this side. + func probeLivenessOnce(completion: @escaping (Bool) -> Void) { + guard connectionState == .connected, !exited else { + completion(false) return } - livenessProbeOutstanding = true - // A probe that cannot even be enqueued means the stream is already unusable. - let enqueued = probeLiveness { [weak self] answered in - self?.livenessProbeOutstanding = false - guard let self else { return } - guard generation == self.processGeneration else { - // A respawn overtook this probe; its answer says nothing about the live stream. - completion?(true) - return - } - if answered { - // The stream is carrying the protocol, so any run of deferred ticks is over. - self.livenessDeferralCount = 0 - completion?(true) - } else { - self.recoverFromStalledTransport() - completion?(false) - } - } - if !enqueued { - livenessProbeOutstanding = false - recoverFromStalledTransport() - completion?(false) - } - } - - /// Asks out of band whether the far end is reachable, then either recovers the stream or - /// defers the verdict to the next tick. See ``checkLivenessAndRecoverIfStalled(completion:)`` - /// for why the silent stream cannot answer this itself. - private func resolveSuspectedStall(generation: UInt64, completion: ((Bool) -> Void)?) { - // One query at a time. The one-shot bounds itself with ssh's `ConnectTimeout` and - // `ServerAlive*` rather than a deadline of its own, so on a dead network it can outlast a - // tick; a second query would ask about the same outage and could recover twice for one - // stall. Report `true` — this tick found nothing wedged, and the query already running is - // the one that decides. - guard !livenessReachabilityQueryInFlight else { - completion?(true) - return - } - livenessReachabilityQueryInFlight = true - // Read the session name now: a `rename-session` during the query would otherwise change - // which session the answer is about. - let reachability = sessionReachability - let host = self.host - let sessionName = self.sessionName - Task { @MainActor [weak self] in - let reachable = await reachability(host, sessionName) - guard let self else { return } - self.livenessReachabilityQueryInFlight = false - // A respawn overtook the query: the stream this answer describes is gone, so acting on - // it would recover a stream that was already replaced. - guard generation == self.processGeneration, self.connectionState == .connected else { - completion?(true) - return - } - // The probe came back while the query was out. The stream is answering after all, so - // there is nothing to recover regardless of what the host said. - guard self.livenessProbeOutstanding else { - completion?(true) - return - } - if reachable { - self.recoverFromStalledTransport() - completion?(false) - return - } - self.livenessDeferralCount += 1 - if self.livenessDeferralCount > Self.maxConsecutiveLivenessDeferrals { - // Long enough. Either the outage is not what is keeping the stream quiet, or it - // has lasted past anything this transport is going to recover from on its own. - self.record("liveness-deferral-exhausted") - self.recoverFromStalledTransport() - completion?(false) - return - } - self.record("liveness-deferred-unreachable") - completion?(true) - } - } - - /// Clears the probe bookkeeping so a fresh stream is judged on its own evidence. - /// - /// Leaves ``livenessReachabilityQueryInFlight`` alone deliberately: a query that is still - /// running clears that itself when it lands, and clearing it here would let a second query - /// start while the first is outstanding. - func resetLivenessProbeState() { - livenessProbeOutstanding = false - livenessDeferralCount = 0 - } - - /// Replaces a transport that is alive but no longer carrying the protocol. - /// - /// Recovery is a respawn rather than an end: the remote session is very likely still there — - /// it is the client that is wedged — so the mirror should be reconnected, not torn down. This - /// routes through the same `beginReconnecting()` path as an ssh transport loss so there is one - /// reconnect implementation rather than a second one for this case. - private func recoverFromStalledTransport() { - guard connectionState == .connected else { return } - record("liveness-stalled") - beginReconnecting() + if !probeLiveness(completion: completion) { completion(false) } } func failPendingTrackedSends() { diff --git a/Sources/RemoteTmuxControlConnection.swift b/Sources/RemoteTmuxControlConnection.swift index fb6b00db00a9..e86b90d29082 100644 --- a/Sources/RemoteTmuxControlConnection.swift +++ b/Sources/RemoteTmuxControlConnection.swift @@ -158,34 +158,6 @@ final class RemoteTmuxControlConnection { /// attempts); cancelled on `stop()` / genuine end so a dead connection stops /// retrying. private var reconnectTask: Task? - /// Periodic liveness probe for transports that reconnect internally (see - /// ``checkLivenessAndRecoverIfStalled(completion:)``). Nil for ssh, which gets an EOF instead. - private var livenessTask: Task? - /// Whether a liveness probe is still waiting for its answer. The next probe's due time is the - /// previous one's deadline, so this is what turns "no answer" into a suspected stall. - var livenessProbeOutstanding = false - /// Whether an out-of-band reachability query is still running. One at a time: the query rides - /// ssh, which bounds itself with `ConnectTimeout`/`ServerAlive*` rather than a deadline of its - /// own, so on a dead network it can outlast a tick. A second query would answer about the same - /// outage and could recover twice for one stall, so a tick that lands on top of one in flight - /// does nothing and lets the query it is waiting on decide. - var livenessReachabilityQueryInFlight = false - /// Consecutive ticks that found the stream silent AND the host unreachable. Reset when a probe - /// is answered and when the connection reconnects, so only an unbroken run counts. - var livenessDeferralCount = 0 - /// How often to ask a self-reconnecting transport whether it is still carrying the protocol. - /// Long enough that an ordinary reconnect finishes untouched, short enough that a wedged - /// mirror is not left silently frozen. - static var livenessProbeIntervalSeconds: UInt64 = 30 - /// How many consecutive unreachable ticks may be deferred before the stream is recovered - /// anyway. At the 30-second interval that is about two minutes of outage. - /// - /// The cap is what keeps deferral honest. A transport that is mid-reconnect deserves to be - /// left alone, but "unreachable" cannot be trusted forever: if the host is both unreachable - /// and the stream is wedged, deferring without a limit leaves a frozen mirror that never - /// retries — the exact failure this monitor was added to prevent. Recovering after the cap - /// costs a reconnect that the backoff would have performed anyway. - static let maxConsecutiveLivenessDeferrals = 4 /// Number of reconnect attempts since the last successful connect, driving the /// capped exponential backoff. Reset to 0 on a successful connect. private var reconnectAttemptCount = 0 @@ -364,44 +336,15 @@ final class RemoteTmuxControlConnection { /// overrides it (which tests do, to assert argv without spawning anything). let transportProfile: RemoteTmuxTransportProfile - /// Asks whether the session is still reachable, on a channel that does not run through this - /// connection's control stream — the one question that separates a wedged stream from a - /// transport that is busy reconnecting underneath. Supplied by the caller the same way - /// ``transportProfile`` is, defaulting to ``oneShotSessionReachability`` (which tests replace, - /// to decide the answer without a host). - let sessionReachability: @Sendable (RemoteTmuxHost, String) async -> Bool - - /// The production reachability check: one `tmux has-session` over ssh's shared control master. - /// - /// Independent of the wedged stream by construction. Even for an et connection, one-shot - /// commands ride ssh (see ``RemoteTmuxETTransportProfile/oneShotArgv(host:remoteCommand:)``), - /// so this asks over a different transport entirely — and against et 6.2.11+7 that difference - /// is measurable: restarting `etserver` closes the control stream while `has-session` keeps - /// succeeding. The master is already open and single-flighted, so the question costs a - /// round-trip and no authentication. - /// - /// Anything other than a clean exit counts as unreachable, including a session tmux says is - /// gone. That is deliberately coarse: a gone session normally arrives as `%exit` on the - /// control stream, and if it somehow does not, the deferral cap recovers within about two - /// minutes and the reattach classifies the end the way it always does. - static let oneShotSessionReachability: @Sendable (RemoteTmuxHost, String) async -> Bool = { - host, sessionName in - let transport = RemoteTmuxSSHTransport(host: host) - let result = try? await transport.runTmux(["has-session", "-t", sessionName]) - return result?.succeeded == true - } - init( host: RemoteTmuxHost, sessionName: String, createIfMissing: Bool = false, pendingPaneSeedByteLimit: Int = RemoteTmuxControlConnection.maximumPendingPaneSeedBytes, - transportProfile: RemoteTmuxTransportProfile? = nil, - sessionReachability: (@Sendable (RemoteTmuxHost, String) async -> Bool)? = nil + transportProfile: RemoteTmuxTransportProfile? = nil ) { self.transportProfile = transportProfile ?? host.transport.profile(port: host.transportPort, terminalPath: host.transportTerminalPath) - self.sessionReachability = sessionReachability ?? Self.oneShotSessionReachability self.host = host self.sessionName = sessionName self.createIfMissing = createIfMissing @@ -628,9 +571,6 @@ final class RemoteTmuxControlConnection { failPendingCommandTransactions() reconnectTask?.cancel() reconnectTask = nil - livenessTask?.cancel() - livenessTask = nil - resetLivenessProbeState() resetWindowListRequestCoalescing() cancelSizingFollowUps() pendingPostAttachAction = nil @@ -669,10 +609,78 @@ final class RemoteTmuxControlConnection { stderrPipeReader = nil stdinWriter?.close() stdinWriter = nil - process?.terminate() + terminateProcessTree(process) process = nil } + /// Ends a spawned transport and everything it started. + /// + /// `Process.terminate()` signals one pid, and a transport is rarely one process: cmux may + /// launch a pty allocator that execs a broker that finally execs the client. Signalling only + /// the allocator leaves the broker and client running, and because the client holds the + /// remote end open they keep their session too — two such trees were found alive hours after + /// their respawns, each still holding a control client on the remote server. + /// + /// Signalling the allocator's process GROUP does not fix it either: `/usr/bin/script` puts + /// its command in a group of its own (measured — the allocator and its payload had different + /// pgids, and the payload survived a group kill). So the tree is walked instead, children + /// before parents, and each process that leads its own group takes that group with it. + /// + /// SIGTERM first because these clients close their remote end on it; anything still alive a + /// moment later is sent SIGKILL, so a client that ignores the polite signal cannot outlive + /// the stream that owns it. + private func terminateProcessTree(_ proc: Process?) { + guard let proc, proc.processIdentifier > 0 else { return } + let root = proc.processIdentifier + let tree = Self.processTree(root: root) + for pid in tree.reversed() { Self.signalProcess(pid, SIGTERM) } + proc.terminate() + Task.detached { + try? await Task.sleep(nanoseconds: 2_000_000_000) + for pid in tree.reversed() where Darwin.kill(pid, 0) == 0 { + Self.signalProcess(pid, SIGKILL) + } + } + } + + /// `root` and its descendants, parents before children, bounded in depth so a pathological + /// tree cannot make teardown expensive. + nonisolated static func processTree(root: pid_t, childrenOf: (pid_t) -> [pid_t] = childPIDs) -> [pid_t] { + var out: [pid_t] = [root] + var frontier = [root] + for _ in 0..<4 { + let next = frontier.flatMap(childrenOf).filter { !out.contains($0) } + if next.isEmpty { break } + out.append(contentsOf: next) + frontier = next + } + return out + } + + /// Sends `signal` to `pid`, and to its process group when `pid` leads one. A leader's group + /// holds the processes it started that the walk cannot see (anything spawned between the + /// listing and the signal). + nonisolated private static func signalProcess(_ pid: pid_t, _ signal: Int32) { + guard pid > 1 else { return } + if getpgid(pid) == pid { _ = Darwin.kill(-pid, signal) } + _ = Darwin.kill(pid, signal) + } + + /// Direct children of `pid`, via the kernel process table (no subprocess, so teardown does + /// not spawn anything while it is tearing down). + nonisolated static let childPIDs: (pid_t) -> [pid_t] = { parent in + var name: [Int32] = [CTL_KERN, KERN_PROC, KERN_PROC_ALL, 0] + var length = 0 + guard sysctl(&name, 4, nil, &length, nil, 0) == 0, length > 0 else { return [] } + let count = length / MemoryLayout.stride + var procs = [kinfo_proc](repeating: kinfo_proc(), count: count) + guard sysctl(&name, 4, &procs, &length, nil, 0) == 0 else { return [] } + let actual = length / MemoryLayout.stride + return procs[0.. RemoteTmuxControlConnection { - RemoteTmuxControlConnection( - host: RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2039), - sessionName: "work", - sessionReachability: { _, _ in reachable } + /// The replacement signal, and the only one: the client exits, which arrives as end of + /// stream and reconnects exactly as it does for ssh. + @Test func aSelfReconnectingTransportRecoversWhenItsClientExits() { + #expect( + RemoteTmuxStreamEndDisposition.forStreamEnd(hasReachedControlMode: true) == .reconnect ) } - /// Runs one monitor tick and waits for its verdict. The verdict can now cross an `await` (the - /// out-of-band question), so it is read from the completion rather than after the call. - @MainActor private static func tick(_ connection: RemoteTmuxControlConnection) async -> Bool { - await withCheckedContinuation { continuation in - connection.checkLivenessAndRecoverIfStalled { continuation.resume(returning: $0) } - } - } - - /// A stream that never reached control mode is a failed start, not a lost session. - /// - /// Reconnecting there is what made a real error — "tmux control stream ended before attach" — - /// surface as an opaque 60-second attach timeout, which cost most of a debugging session. - /// Reconnecting is right only once a stream has actually worked. @Test func endOfStreamBeforeControlModeIsTerminalRatherThanRetried() { #expect( RemoteTmuxStreamEndDisposition.forStreamEnd(hasReachedControlMode: false) == .sessionOver, @@ -527,18 +473,22 @@ import Testing ) } - /// ssh must be untouched by all of this: it gets an EOF, `handleStreamEnd` already recovers, - /// and probing an idle ssh stream would add traffic and a new way to fail. - @MainActor @Test func anSSHTransportIsNeverProbedForStalls() { - let connection = RemoteTmuxControlConnection( - host: RemoteTmuxHost(destination: "user@host"), - sessionName: "work" - ) - connection.handle(.enter) - var reported: Bool? - connection.checkLivenessAndRecoverIfStalled { reported = $0 } - #expect(reported == true, "ssh is out of scope for the stall check") - #expect(!connection.snapshot().recentEvents.contains("liveness-stalled")) + /// Teardown has to reach the whole transport, not just the process cmux launched. A pty + /// allocator execs a broker that execs the client, and `/usr/bin/script` puts that payload + /// in a process group of its own — so signalling the allocator, or its group, leaves the + /// client running and still attached to the remote server. + @Test func teardownWalksTheWholeTransportTree() { + // allocator 100 -> broker 200 -> client 300, with an unrelated 999 that must be left alone. + let children: [pid_t: [pid_t]] = [100: [200], 200: [300], 999: [1000]] + let tree = RemoteTmuxControlConnection.processTree(root: 100) { children[$0] ?? [] } + #expect(tree == [100, 200, 300]) + #expect(!tree.contains(999) && !tree.contains(1000)) + } + + /// A cycle or a runaway tree must not make teardown unbounded. + @Test func theTreeWalkIsBounded() { + let tree = RemoteTmuxControlConnection.processTree(root: 1) { [$0 + 1] } + #expect(tree.count <= 5, "the walk stops at its depth bound, got \(tree.count)") } /// The client binary is resolved, not assumed. A literal `/usr/local/bin/et` is a claim about From 51d9d308abd13e501add02f0a2e2eda9a1eebcaa Mon Sep 17 00:00:00 2001 From: ejc3 Date: Mon, 31 Aug 2026 11:06:11 -0700 Subject: [PATCH 20/23] remote-tmux: document what replaced the stall detector MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The seam design still described the probe-and-respawn monitor as the plan, and the no-polling lint still carried its exemption. Both now say what the code does: a transport that reconnects internally is left alone, its client's exit is the only trigger, and the one uncovered case is a far end that is suspended rather than dead — measured on a local et rig, where killing tmux produced %exit and an exiting client, while a SIGSTOPped etterminal produced nothing at all. The lint's new exemption is the SIGKILL escalation in teardown: a process that ignores SIGTERM emits no event, so the absence of an exit is only observable by looking again. --- docs/remote-tmux-transport-seam.md | 54 ++++++++++++++------------ scripts/lint-remote-tmux-no-polling.sh | 2 +- 2 files changed, 30 insertions(+), 26 deletions(-) diff --git a/docs/remote-tmux-transport-seam.md b/docs/remote-tmux-transport-seam.md index 9296ba8a1bbf..6e79e2a9ff45 100644 --- a/docs/remote-tmux-transport-seam.md +++ b/docs/remote-tmux-transport-seam.md @@ -97,28 +97,32 @@ therefore correct and cmux notices nothing, which is the goal. What it must not treat a *stall* as death and respawn on a timer, because that discards a session the transport was about to recover. -So the failure mode moves from "stream ended" to "stream is alive but wedged", and a -transport that reconnects internally needs a liveness check instead: - -1. the transport process is still alive, and -2. a control-mode round-trip completes. - -cmux already has the round-trip primitive: a bounded `display-message -p ok` query, used as -`awaitCommandBarrier` in `RemoteTmuxViewConnection`. Reuse it rather than inventing a -heartbeat, and give the probe a deadline: measured against 6.2.11+7, et can accept stdin while -producing no control output, so an unanswered probe is the suspicion. The next probe's due time -is that deadline, which needs no second clock. - -An unanswered probe cannot be the verdict, though, because a real network interruption looks -identical from the stream's side: the transport is reconnecting underneath and cannot answer -either, and recovering there kills the process and throws away the session it was resuming. -What tells the two apart is a question asked somewhere else. One-shot commands ride ssh's -shared master even for an et connection, so `tmux has-session` reaches the host over a channel -this stream's wedge cannot touch — the same asymmetry the `etserver` restart above measures. A -host that answers proves the stream is the broken part, and that is the case to recover. A host -that does not answer is an outage, so stay connected and ask again next tick, capped at four -consecutive deferrals (about two minutes) so a host that is both unreachable and wedged still -gets its reconnect. +So the tempting next step is a liveness check: probe the stream on a timer and respawn it +when a probe goes unanswered. cmux shipped that and then removed it, because the cure was +worse than the disease. + +An unanswered probe cannot be the verdict. A real network interruption looks identical from +the stream's side — the transport is reconnecting underneath and cannot answer either — and +recovering there kills the process that was about to recover on its own. Asking the question +somewhere else does not save it: the out-of-band answer says whether the HOST is reachable, +not whether this client is wedged, and a host that answers while a client is mid-reconnect +reads as "the stream is the broken part" when it is not. + +Measured overnight against a real host: the timer respawned a healthy et client twice, and +because a new et session bootstraps over ssh, each replacement hit an interactive second +factor with nobody there to answer it. The mirror came back dead with two orphaned client +trees behind it. A transport that reconnects internally is therefore left alone; its own exit +is the only trigger, the same end-of-stream signal ssh uses. + +That is safe because death is loud. Measured on a local et rig: killing the tmux server +emitted `%exit`, then `Session terminated`, and the et client process exited — every link +holds a descriptor on the one below it, so an ordinary death propagates all the way up. The +one case with no signal is a far end that is suspended rather than dead: a `SIGSTOP`ped +`etterminal` keeps its descriptors open, keepalives cover only client↔etserver, and nothing +reports anything. That case is left uncovered on purpose. A suspended process resumes with +its session intact, so respawning would destroy work that was coming back; if it ever needs +covering, the treatment is a visible stale state with a manual reconnect, never an automatic +respawn. A transport exit does **not** mean the session is over, tempting as the symmetry is. Restarting only `etserver` ends the stream while `tmux has-session` still succeeds, so acting on it discarded @@ -310,8 +314,8 @@ than rediscovering it after a transport swap. behavior change, so the existing suites are the regression gate. 3. Add the fuzz harness against the mock transport, with the invariants above. It should pass for `SSHTransport` before any new transport exists. -4. Add the persistent-session transport behind per-host opt-in, with the liveness check and - a fallback to `SSHTransport` when the transport binary is missing. Turn on the - internally-reconnecting arm of the fuzz. +4. Add the persistent-session transport behind per-host opt-in, recovering only on its + client's exit, with a fallback to `SSHTransport` when the transport binary is missing. + Turn on the internally-reconnecting arm of the fuzz. 5. Add the real sshd plus upstream-ET end-to-end fixture, including a real network drop. 6. Add the pre-connect hook, with the race and timeout cases in the fuzz. diff --git a/scripts/lint-remote-tmux-no-polling.sh b/scripts/lint-remote-tmux-no-polling.sh index c41abe576216..9f043621eb79 100755 --- a/scripts/lint-remote-tmux-no-polling.sh +++ b/scripts/lint-remote-tmux-no-polling.sh @@ -42,7 +42,7 @@ BASELINE_FILE="scripts/remote-tmux-polling-baseline.txt" # Exceptions introduced deliberately, with the reason no edge exists. ALLOW=( - "Sources/RemoteTmuxControlConnection.swift:startLivenessMonitorIfNeeded|A transport that reconnects internally emits NO EOF for a network drop: its process stays up and the stream pauses, so a wedged transport is indistinguishable from an idle one. There is no event for 'still carrying the protocol' — the only way to know is to ask, so this probes rather than waits." + "Sources/RemoteTmuxControlConnection.swift:terminateProcessTree|SIGKILL escalation after SIGTERM. The edge would be 'the process handled the signal', and a process that IGNORES SIGTERM emits nothing at all — the absence of an exit is only observable by giving it a moment and looking again." "Sources/RemoteTmuxControlConnection.swift:scheduleReconnectAttempt|Reconnect backoff for a host that is unreachable. The edge would be 'the host came back', which nothing local can observe; retrying IS the observation." ) From 6761fe4f310044268f424dcf6ea3e1af3a5498e6 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Mon, 31 Aug 2026 11:39:27 -0700 Subject: [PATCH 21/23] remote-tmux: drop the transport properties nothing reads any more MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `reconnectsInternally` existed to route a transport to the stall monitor, and `probeLiveness` existed to feed that monitor a round trip. With the monitor gone, the flag is declared, implemented twice and read nowhere, and the probe is called only by a test calling it — a protocol requirement every future transport would have to answer for nobody, and a pair of functions kept alive by their own test. The rule they used to express is now enforced somewhere better: `scripts/lint-remote-tmux-no-polling.sh` fails on a timer in these sources, so a reintroduced probe loop is a failing lint rather than a property that quietly changes meaning. --- ...RemoteTmuxControlConnection+Commands.swift | 48 ------------- Sources/RemoteTmuxTransportRegistry.swift | 12 ---- .../RemoteTmuxProxyTransportRetryTests.swift | 71 ++----------------- 3 files changed, 4 insertions(+), 127 deletions(-) diff --git a/Sources/RemoteTmuxControlConnection+Commands.swift b/Sources/RemoteTmuxControlConnection+Commands.swift index cc25f7fd9d01..cd4af61ad56e 100644 --- a/Sources/RemoteTmuxControlConnection+Commands.swift +++ b/Sources/RemoteTmuxControlConnection+Commands.swift @@ -85,54 +85,6 @@ extension RemoteTmuxControlConnection { return true } - /// Asks tmux to answer, so a stalled-but-alive transport can be told from a healthy one. - /// - /// This is the liveness check a transport that owns its own reconnection needs. cmux's - /// recovery is built on stdout EOF, but such a transport produces no EOF for a network - /// drop — the stream pauses and resumes — so EOF cannot be the trigger and a stall must - /// not be mistaken for death. What is left is asking the far end a question: - /// - /// - the process is still alive, and - /// - a control-mode round-trip completes. - /// - /// `display-message -p` is the cheapest question that proves both. It is a read, so it - /// moves no client size and mutates nothing, and it resolves through the same - /// `%begin`/`%end` correlation as any other command — which is why this reuses - /// ``sendTracked(_:completion:)`` rather than inventing a heartbeat with its own timer - /// and its own failure modes. - /// - /// - Parameter completion: `true` when tmux answered, `false` when the block resolved as - /// an error or the stream reset before answering. Not called at all if the command - /// could not be enqueued, which the `false` return reports. - @discardableResult - func probeLiveness(completion: @escaping (Bool) -> Void) -> Bool { - guard !exited else { return false } - return sendTracked("display-message -p cmux-liveness", completion: completion) - } - - /// Whether the stream is answering, asked once on demand. - /// - /// There is deliberately no periodic version of this and no recovery attached to it. cmux - /// used to probe a self-reconnecting transport every 30 seconds and respawn it when a probe - /// went unanswered, on the theory that such a transport can be alive but wedged. Measured - /// against et on a host that requires a second factor, the cure was far worse: an et client - /// riding out a network change is quiet for longer than a tick, so the probe killed the one - /// process that could have recovered silently, and its replacement had to bootstrap a new - /// session over ssh — which needs an interactive second factor nobody is there to give at - /// 03:00. The mirror came back dead with two orphaned clients behind it. - /// - /// et answers this itself: its client exchanges keepalives every few seconds and exits when - /// they stop, so a quiet client is idle or reconnecting, never wedged, and its exit already - /// reaches ``handleStreamEnd(processGeneration:)`` like any other end of stream. Trusting - /// that is both simpler and strictly better informed than guessing from this side. - func probeLivenessOnce(completion: @escaping (Bool) -> Void) { - guard connectionState == .connected, !exited else { - completion(false) - return - } - if !probeLiveness(completion: completion) { completion(false) } - } - func failPendingTrackedSends() { let completions = Array(trackedSendCompletions.values) trackedSendCompletions.removeAll() diff --git a/Sources/RemoteTmuxTransportRegistry.swift b/Sources/RemoteTmuxTransportRegistry.swift index a9572f2192b4..04d5dddc334f 100644 --- a/Sources/RemoteTmuxTransportRegistry.swift +++ b/Sources/RemoteTmuxTransportRegistry.swift @@ -86,15 +86,6 @@ protocol RemoteTmuxTransportProfile: Sendable { /// whether a transport can be spawned at all, independent of argv. var requiresPseudoTerminal: Bool { get } - /// Whether the transport recovers from network loss by itself. - /// - /// This is the property that changes cmux's behavior rather than just its argv. cmux - /// treats stdout EOF as "the stream died, respawn with backoff". A transport that - /// reconnects internally produces no EOF for a network drop — the stream pauses and - /// resumes — so respawning on a stall would throw away the session it was about to - /// recover. Such a transport needs a liveness check (process alive plus a control-mode - /// round-trip) instead of an EOF trigger. - var reconnectsInternally: Bool { get } } /// What end-of-stream on the control connection means. @@ -193,8 +184,6 @@ struct RemoteTmuxSSHTransportProfile: RemoteTmuxTransportProfile { + ["--", host.destination, remoteCommand] } - /// ssh does not: a dropped connection ends the process, and cmux respawns it. - var reconnectsInternally: Bool { false } /// ssh is happy on pipes, which is what cmux spawns today. var requiresPseudoTerminal: Bool { false } @@ -403,7 +392,6 @@ struct RemoteTmuxETTransportProfile: RemoteTmuxTransportProfile { RemoteTmuxSSHTransportProfile().oneShotArgv(host: host, remoteCommand: remoteCommand) } - var reconnectsInternally: Bool { true } var requiresPseudoTerminal: Bool { true } } diff --git a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift index 43a5bd7e9247..2a9fb91b1991 100644 --- a/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift +++ b/cmuxTests/RemoteTmuxProxyTransportRetryTests.swift @@ -130,7 +130,6 @@ import Testing RemoteTmuxSSHTransportProfile().oneShotArgv(host: host, remoteCommand: remoteCommand) } - var reconnectsInternally: Bool { true } /// A persistent-session transport is a terminal client, so it needs a tty. var requiresPseudoTerminal: Bool { true } } @@ -160,7 +159,6 @@ import Testing /// ssh owns no reconnection: cmux respawns it. This is the flag that decides whether EOF /// or a liveness check drives recovery, so it is worth pinning rather than assuming. @Test func sshDoesNotReconnectItself() { - #expect(!RemoteTmuxSSHTransportProfile().reconnectsInternally) } /// The seam has to be able to express a transport that is not ssh at all: a different @@ -171,7 +169,6 @@ import Testing let argv = profile.controlStreamArgv(host: host, sessionName: "work", createIfMissing: false) #expect(profile.executablePath() == "/usr/local/bin/et") - #expect(profile.reconnectsInternally) #expect(argv.last == "user@host") #expect(consecutive(argv, "--port", "2022")) // The command must run one command rather than opening a shell, and must still go @@ -229,7 +226,6 @@ import Testing host: RemoteTmuxHost(destination: "user@host"), sessionName: "work" ) - #expect(!connection.transportProfile.reconnectsInternally) #expect(connection.transportProfile.executablePath() == RemoteTmuxHost.defaultSSHExecutablePath()) } } @@ -411,49 +407,11 @@ import Testing ) } - /// cmux does not second-guess a transport that owns its reconnection. + /// cmux does not second-guess a transport that owns its reconnection: there is no probe + /// and no timer, so a client that goes quiet while it reconnects is simply left alone. + /// `scripts/lint-remote-tmux-no-polling.sh` is what keeps it that way — a reintroduced + /// timer fails that lint rather than needing a test that waits one out here. /// - /// It used to: a 30-second probe, and a respawn when one went unanswered. An et client - /// riding out a network change is quiet for longer than that, so the probe killed the - /// process that was about to recover on its own, and the replacement had to bootstrap a - /// new session — which on a host with a second factor cannot happen unattended. What was - /// left was a dead mirror and orphaned clients. - /// - /// et exchanges keepalives every few seconds and exits when they stop, so its own exit is - /// both the earlier and the better-informed signal, and it already arrives as end of stream. - @MainActor @Test func aQuietSelfReconnectingTransportIsLeftAlone() { - let connection = RemoteTmuxControlConnection( - host: RemoteTmuxHost(destination: "user@host", transport: .et, transportPort: 2039), - sessionName: "work" - ) - // A real stream to write into, so the probe is genuinely SENT and then genuinely goes - // unanswered — the shape of an et client mid-reconnect. Without a writer the probe - // merely fails to enqueue, which proves nothing about the policy under test. - let pipe = Pipe() - let writer = RemoteTmuxControlPipeWriter( - handle: pipe.fileHandleForWriting, - label: "et-quiet-stream-test", - maxPendingBytes: 1 << 16, - onFailure: {} - ) - defer { writer.close(); try? pipe.fileHandleForReading.close() } - connection.installStdinWriterForTesting(writer) - connection.handle(.enter) - #expect(connection.connectionState == .connected) - - var reported: Bool? - connection.probeLivenessOnce { reported = $0 } - #expect(reported == nil, "a probe that was sent stays outstanding until tmux answers") - // A second ask on top of an unanswered one is precisely what the deleted detector - // treated as a stall: probe N had to be answered before probe N+1 was due, and missing - // that deadline respawned the stream. Asking twice here is what makes this test fail if - // that policy ever comes back. - connection.probeLivenessOnce { reported = $0 } - #expect(connection.connectionState == .connected, "a quiet client must not be recovered") - #expect(!connection.snapshot().recentEvents.contains("liveness-stalled")) - #expect(!connection.snapshot().recentEvents.contains("reconnecting")) - } - /// The replacement signal, and the only one: the client exits, which arrives as end of /// stream and reconnects exactly as it does for ssh. @Test func aSelfReconnectingTransportRecoversWhenItsClientExits() { @@ -669,28 +627,11 @@ import Testing /// The two properties that decide behavior rather than argv. @Test func etOwnsItsReconnectionAndNeedsATTY() { let profile = RemoteTmuxETTransportProfile() - #expect(profile.reconnectsInternally) #expect(profile.requiresPseudoTerminal) } // MARK: - Seam 2: telling a stall from a death - /// A transport that owns its reconnection needs a question it can answer, because EOF is - /// no longer the signal: it does not end for a network drop. The probe must be a read - /// (mutating nothing, moving no client size) and must resolve through the same - /// `%begin`/`%end` correlation as any other command rather than a bespoke heartbeat. - @MainActor @Test func alivenessIsProvedByAControlModeRoundTrip() { - let connection = RemoteTmuxControlConnection( - host: RemoteTmuxHost(destination: "user@host", transport: .et), - sessionName: "work" - ) - // Never started, so there is no stream to carry a question: the caller is told the - // probe did not leave, rather than being left waiting on a completion that cannot come. - var completionFired = false - let enqueued = connection.probeLiveness { _ in completionFired = true } - #expect(!enqueued, "a probe must report that it could not be sent on a dead stream") - #expect(!completionFired, "no completion may fire for a probe that never left") - } /// The property that makes the probe necessary in the first place. /// @@ -701,8 +642,6 @@ import Testing /// ended (see endOfStreamAlwaysMeansReconnectAndLetTheReattachDecide). @Test func aStallIsNotADeathForATransportThatReconnectsItself() { let et = RemoteTmuxETTransportProfile() - #expect(et.reconnectsInternally) - #expect(!RemoteTmuxSSHTransportProfile().reconnectsInternally) } // MARK: - Runtime selection @@ -716,7 +655,6 @@ import Testing let etHost = RemoteTmuxHost(destination: "user@host", transport: .et) let profile = etHost.transport.profile(port: etHost.port) #expect(profile.requiresPseudoTerminal) - #expect(profile.reconnectsInternally) } /// The transport's port is NOT ssh's port, and conflating them breaks every one-shot. @@ -841,7 +779,6 @@ private struct SplitMix64 { func aTransportThatOwnsReconnectionIsNeverRespawnedForAStall(seed: UInt64) { var rng = SplitMix64(seed: seed) let profile = RemoteTmuxETTransportProfile() - #expect(profile.reconnectsInternally, "this suite is about internally-reconnecting transports") var model = Model() From 5923f349b0ff02b206ad53e6dd7ba0b4a238c0cc Mon Sep 17 00:00:00 2001 From: ejc3 Date: Mon, 14 Sep 2026 12:37:26 -0700 Subject: [PATCH 22/23] remote-tmux: list the pane-seed delivery deadline the polling guard flags --- scripts/lint-remote-tmux-no-polling.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/lint-remote-tmux-no-polling.sh b/scripts/lint-remote-tmux-no-polling.sh index 9f043621eb79..cca013174b65 100755 --- a/scripts/lint-remote-tmux-no-polling.sh +++ b/scripts/lint-remote-tmux-no-polling.sh @@ -44,6 +44,7 @@ BASELINE_FILE="scripts/remote-tmux-polling-baseline.txt" ALLOW=( "Sources/RemoteTmuxControlConnection.swift:terminateProcessTree|SIGKILL escalation after SIGTERM. The edge would be 'the process handled the signal', and a process that IGNORES SIGTERM emits nothing at all — the absence of an exit is only observable by giving it a moment and looking again." "Sources/RemoteTmuxControlConnection.swift:scheduleReconnectAttempt|Reconnect backoff for a host that is unreachable. The edge would be 'the host came back', which nothing local can observe; retrying IS the observation." + "Sources/RemoteTmuxSessionMirror+OutputRouting.swift:schedulePaneSeedDeliveryDeadline|Deadline arm on a pane's readiness wait: the task is cancelled when the surface becomes ready, and on expiry the seed is drained or gracefully deferred rather than retried" ) fail=0 From 4e3af4a15a3370fce1f21b9d0307e0aedf3d8954 Mon Sep 17 00:00:00 2001 From: ejc3 Date: Mon, 14 Sep 2026 12:57:45 -0700 Subject: [PATCH 23/23] localization: retranslate the ssh-tmux help for the transport flags --- Resources/Localizable.xcstrings | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/Resources/Localizable.xcstrings b/Resources/Localizable.xcstrings index db91e796584e..0351a4ba23b0 100644 --- a/Resources/Localizable.xcstrings +++ b/Resources/Localizable.xcstrings @@ -87285,8 +87285,8 @@ "localizations": { "ar": { "stringUnit": { - "state": "needs_review", - "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nاعكس جلسات tmux للمضيف البعيد داخل نافذة cmux الحالية عبر وضع تحكم tmux بواسطة SSH. تصبح كل جلسة مساحة عمل، وكل نافذة علامة تبويب، وكل نافذة متعددة الأجزاء تقسيمًا أصليًا. يتطلب ذلك تفعيل الإصدار التجريبي \"Remote tmux\".\n\nعند الحاجة إلى مصادقة تفاعلية، يشغّل cmux أمر ssh في هذه الطرفية ثم يعيد المحاولة عبر الاتصال المشترك. تُحترم الأسماء المستعارة وإعدادات ~/.ssh/config.\n\nFlags:\n --port منفذ SSH\n --identity مسار ملف هوية SSH\n --no-focus عدم تحديد مساحة العمل المعكوسة أو تنشيط نافذتها\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" + "state": "translated", + "value": "Usage: cmux ssh-tmux [--port ] [--identity ]\n [--transport ssh|et] [--transport-port ] [--no-focus]\n\nيعكس جلسات tmux لمضيف بعيد في الشريط الجانبي للنافذة الحالية عبر وضع\nالتحكم في tmux عبر SSH (tmux -CC). تصبح كل جلسة مساحة عمل، وكل نافذة\nعلامة تبويب، وكل نافذة متعددة الأجزاء تقسيمًا أصليًا. يتطلب تفعيل\nإعداد الميزة التجريبية \"Remote tmux\".\n\nإذا احتاج المضيف إلى مصادقة تفاعلية (كلمة مرور، أو تأكيد مفتاح المضيف، أو MFA،\nأو لمس مفتاح أمان)، يشغّل cmux الأمر ssh مباشرة في هذه الطرفية لتتمكن من\nالمصادقة، ثم يعكس الجلسات عبر اتصال SSH المشترك. أما المضيفون الذين\nتتم مصادقتهم دون تفاعل (ssh-agent أو مفتاح في ~/.ssh/config) فيُعكَسون\nدون أي مطالبة. تُحترم الأسماء المستعارة في ~/.ssh/config وإعدادات IdentityFile/ProxyJump/Port الخاصة بها.\n\nFlags:\n --port منفذ SSH\n --identity مسار ملف هوية SSH\n --transport ssh (الافتراضي) أو et. ينقل et تدفق التحكم عبر\n EternalTerminal الذي يعيد الاتصال تلقائيًا، لذا تبقى\n المرآة قائمة عند تغيّر الشبكة بدلًا من إعادة تشغيلها.\n --transport-port المنفذ الذي يتصل به النقل: منفذ sshd لـ ssh، ومنفذ etserver\n لـ et (الافتراضي 22 لـ ssh و2022 لـ et)\n --no-focus عدم تحديد مساحة العمل المنعكسة أو تنشيط نافذتها\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519\n cmux ssh-tmux dev@my-host --transport et --transport-port 8080" } }, "bs": { @@ -87303,8 +87303,8 @@ }, "de": { "stringUnit": { - "state": "needs_review", - "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nSpiegelt die tmux-Sitzungen eines entfernten Hosts per SSH-tmux-Steuermodus in das aktuelle cmux-Fenster. Jede Sitzung wird zu einem Arbeitsbereich, jedes Fenster zu einem Tab und jedes Fenster mit mehreren Bereichen zu einer nativen Teilung. Erfordert die aktivierte Betaoption \"Remote tmux\".\n\nWenn eine interaktive Authentifizierung nötig ist, führt cmux ssh in diesem Terminal aus und versucht es anschließend über die gemeinsame Verbindung erneut. Aliase und Einstellungen aus ~/.ssh/config werden berücksichtigt.\n\nFlags:\n --port SSH-Port\n --identity Pfad zur SSH-Identitätsdatei\n --no-focus Spiegel-Arbeitsbereich nicht auswählen und Fenster nicht aktivieren\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" + "state": "translated", + "value": "Usage: cmux ssh-tmux [--port ] [--identity ]\n [--transport ssh|et] [--transport-port ] [--no-focus]\n\nSpiegelt die tmux-Sitzungen eines entfernten Hosts über den SSH-tmux-Steuermodus\n(tmux -CC) in die Seitenleiste des aktuellen Fensters. Jede Sitzung wird zu\neinem Arbeitsbereich, jedes Fenster zu einem Tab und jedes Fenster mit mehreren\nBereichen zu einer nativen Teilung. Erfordert die Betaoption \"Remote tmux\".\n\nWenn der Host eine interaktive Authentifizierung braucht (Passwort, Bestätigung\ndes Hostschlüssels, MFA oder Berührung eines Sicherheitsschlüssels), führt cmux\nssh direkt in diesem Terminal aus, damit du dich anmelden kannst, und spiegelt die\nSitzungen dann über die gemeinsame SSH-Verbindung. Hosts ohne interaktive Anmeldung\n(ssh-agent oder Schlüssel in ~/.ssh/config) werden ohne Rückfrage gespiegelt. Aliase aus ~/.ssh/config und deren IdentityFile/ProxyJump/Port-Einstellungen werden berücksichtigt.\n\nFlags:\n --port SSH-Port\n --identity Pfad zur SSH-Identitätsdatei\n --transport ssh (Standard) oder et. et überträgt den Steuerstrom über\n EternalTerminal, das sich selbst neu verbindet, sodass der\n Spiegel einen Netzwechsel übersteht, statt neu zu starten.\n --transport-port Port, mit dem sich der Transport verbindet: der von sshd für ssh,\n der von etserver für et (Standard 22 für ssh, 2022 für et)\n --no-focus Spiegel-Arbeitsbereich nicht auswählen und Fenster nicht fokussieren\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519\n cmux ssh-tmux dev@my-host --transport et --transport-port 8080" } }, "en": { @@ -87315,14 +87315,14 @@ }, "es": { "stringUnit": { - "state": "needs_review", - "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nRefleja las sesiones tmux de un host remoto en la ventana actual de cmux mediante el modo de control tmux por SSH. Cada sesión se convierte en un espacio de trabajo, cada ventana en una pestaña y cada ventana con varios paneles en una división nativa. Requiere activar la función beta \"Remote tmux\".\n\nSi se necesita autenticación interactiva, cmux ejecuta ssh en este terminal y luego reintenta mediante la conexión compartida. Se respetan los alias y ajustes de ~/.ssh/config.\n\nFlags:\n --port puerto SSH\n --identity ruta del archivo de identidad SSH\n --no-focus no seleccionar el espacio reflejado ni activar su ventana\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" + "state": "translated", + "value": "Usage: cmux ssh-tmux [--port ] [--identity ]\n [--transport ssh|et] [--transport-port ] [--no-focus]\n\nRefleja las sesiones de tmux de un host remoto en la barra lateral de la ventana\nactual mediante el modo de control de tmux por SSH (tmux -CC). Cada sesión se\nconvierte en un espacio de trabajo, cada ventana en una pestaña y cada ventana con\nvarios paneles en una división nativa. Requiere el ajuste beta \"Remote tmux\".\n\nSi el host necesita autenticación interactiva (contraseña, confirmación de la clave\ndel host, MFA o tocar una llave de seguridad), cmux ejecuta ssh directamente en este\nterminal para que puedas autenticarte y luego refleja las sesiones por la conexión\nSSH compartida. Los hosts que se autentican sin interacción (ssh-agent / clave en\n~/.ssh/config) se reflejan sin preguntar. Se respetan los alias de ~/.ssh/config y sus ajustes IdentityFile/ProxyJump/Port.\n\nFlags:\n --port Puerto SSH\n --identity Ruta del archivo de identidad SSH\n --transport ssh (predeterminado) o et. et lleva el flujo de control por\n EternalTerminal, que se reconecta por sí solo, así que el\n reflejo sobrevive a un cambio de red en lugar de reiniciarse.\n --transport-port Puerto al que se conecta el transporte: el de sshd para ssh,\n el de etserver para et (predeterminado 22 para ssh, 2022 para et)\n --no-focus No seleccionar el espacio de trabajo reflejado ni activar su ventana\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519\n cmux ssh-tmux dev@my-host --transport et --transport-port 8080" } }, "fr": { "stringUnit": { - "state": "needs_review", - "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nReproduit les sessions tmux d’un hôte distant dans la fenêtre cmux actuelle via le mode de contrôle tmux sur SSH. Chaque session devient un espace de travail, chaque fenêtre un onglet et chaque fenêtre à plusieurs volets une division native. Nécessite l’option bêta \"Remote tmux\".\n\nSi une authentification interactive est nécessaire, cmux exécute ssh dans ce terminal puis réessaie via la connexion partagée. Les alias et réglages de ~/.ssh/config sont respectés.\n\nFlags:\n --port port SSH\n --identity chemin du fichier d’identité SSH\n --no-focus ne pas sélectionner l’espace miroir ni activer sa fenêtre\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" + "state": "translated", + "value": "Usage: cmux ssh-tmux [--port ] [--identity ]\n [--transport ssh|et] [--transport-port ] [--no-focus]\n\nReflète les sessions tmux d’un hôte distant dans la barre latérale de la fenêtre\nactuelle via le mode de contrôle tmux de SSH (tmux -CC). Chaque session devient un\nespace de travail, chaque fenêtre un onglet et chaque fenêtre à plusieurs volets\nune division native. Nécessite le réglage bêta « Remote tmux ».\n\nSi l’hôte exige une authentification interactive (mot de passe, confirmation de\nla clé d’hôte, MFA ou contact d’une clé de sécurité), cmux exécute ssh directement\ndans ce terminal pour que vous puissiez vous authentifier, puis reflète les sessions\nvia la connexion SSH partagée. Les hôtes qui s’authentifient sans interaction\n(ssh-agent / clé dans ~/.ssh/config) sont reflétés sans invite. Les alias de ~/.ssh/config et leurs réglages IdentityFile/ProxyJump/Port sont respectés.\n\nFlags:\n --port Port SSH\n --identity Chemin du fichier d’identité SSH\n --transport ssh (par défaut) ou et. et transporte le flux de contrôle via\n EternalTerminal, qui se reconnecte tout seul : le miroir\n survit à un changement de réseau au lieu d’être relancé.\n --transport-port Port auquel le transport se connecte : celui de sshd pour ssh,\n celui d’etserver pour et (22 par défaut pour ssh, 2022 pour et)\n --no-focus Ne pas sélectionner l’espace de travail miroir ni activer sa fenêtre\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519\n cmux ssh-tmux dev@my-host --transport et --transport-port 8080" } }, "it": { @@ -87333,8 +87333,8 @@ }, "ja": { "stringUnit": { - "state": "needs_review", - "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nSSH の tmux コントロールモードで、リモートホストの tmux セッションを現在の cmux ウィンドウにミラーリングします。各セッションはワークスペースに、各ウィンドウはタブに、複数ペインのウィンドウはネイティブな分割になります。「Remote tmux」ベータ設定を有効にする必要があります。\n\n対話型認証が必要な場合、cmux はこのターミナルで ssh を実行し、共有接続を使って再試行します。~/.ssh/config のエイリアスと設定も使用されます。\n\nFlags:\n --port SSH ポート\n --identity SSH 識別ファイルのパス\n --no-focus ミラーワークスペースを選択せず、ウィンドウもアクティブにしない\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" + "state": "translated", + "value": "Usage: cmux ssh-tmux [--port ] [--identity ]\n [--transport ssh|et] [--transport-port ] [--no-focus]\n\nSSH の tmux コントロールモード (tmux -CC) で、リモートホストの tmux セッションを\n現在のウィンドウのサイドバーにミラーリングします。各セッションはワークスペースに、\n各ウィンドウはタブに、複数ペインのウィンドウはネイティブな分割になります。\n「Remote tmux」ベータ設定を有効にする必要があります。\n\nホストが対話型認証 (パスワード、ホストキーの確認、MFA、セキュリティキーのタッチ) を\n必要とする場合、cmux はこのターミナルで ssh を直接実行して認証できるようにし、\nその後、共有 SSH 接続でセッションをミラーリングします。\n非対話で認証できるホスト (ssh-agent や ~/.ssh/config の鍵) は\n確認なしでミラーリングされます。~/.ssh/config のエイリアスと IdentityFile/ProxyJump/Port の設定も使用されます。\n\nFlags:\n --port SSH ポート\n --identity SSH 識別ファイルのパス\n --transport ssh (既定) または et。et はコントロールストリームを\n EternalTerminal 経由で送ります。自動で再接続するため、\n ネットワークが変わってもミラーは再起動せずに維持されます。\n --transport-port トランスポートの接続先ポート。ssh では sshd、et では etserver\n のポートです (既定は ssh が 22、et が 2022)\n --no-focus ミラーワークスペースを選択せず、ウィンドウもアクティブにしない\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519\n cmux ssh-tmux dev@my-host --transport et --transport-port 8080" } }, "km": { @@ -87345,8 +87345,8 @@ }, "ko": { "stringUnit": { - "state": "needs_review", - "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\nSSH tmux 제어 모드로 원격 호스트의 tmux 세션을 현재 cmux 창에 미러링합니다. 각 세션은 작업 공간이 되고, 각 창은 탭이 되며, 여러 패널이 있는 창은 네이티브 분할이 됩니다. \"Remote tmux\" 베타 설정을 켜야 합니다.\n\n대화형 인증이 필요하면 cmux가 이 터미널에서 ssh를 실행한 뒤 공유 연결을 통해 다시 시도합니다. ~/.ssh/config의 별칭과 설정도 적용됩니다.\n\nFlags:\n --port SSH 포트\n --identity SSH ID 파일 경로\n --no-focus 미러 작업 공간을 선택하거나 창을 활성화하지 않음\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" + "state": "translated", + "value": "Usage: cmux ssh-tmux [--port ] [--identity ]\n [--transport ssh|et] [--transport-port ] [--no-focus]\n\nSSH tmux 제어 모드(tmux -CC)를 통해 원격 호스트의 tmux 세션을 현재 창의\n사이드바에 미러링합니다. 각 세션은 작업 공간, 각 윈도우는 탭, 여러 패널이 있는\n윈도우는 네이티브 분할이 됩니다. \"Remote tmux\" 베타 설정이\n필요합니다.\n\n호스트에 대화형 인증(비밀번호, 호스트 키 확인, MFA, 보안 키 터치)이 필요하면\ncmux가 이 터미널에서 ssh를 직접 실행해 인증할 수 있게 한 뒤,\n공유 SSH 연결로 세션을 미러링합니다. 대화 없이 인증되는 호스트\n(ssh-agent 또는 ~/.ssh/config의 키)는 확인 없이\n미러링됩니다. ~/.ssh/config 별칭과 IdentityFile/ProxyJump/Port 설정이 적용됩니다.\n\nFlags:\n --port SSH 포트\n --identity SSH ID 파일 경로\n --transport ssh(기본값) 또는 et. et는 제어 스트림을\n 스스로 다시 연결하는 EternalTerminal로 전달하므로\n 네트워크가 바뀌어도 미러가 다시 시작되지 않고 유지됩니다.\n --transport-port 전송이 연결할 포트: ssh는 sshd, et는 etserver의 포트\n (기본값: ssh 22, et 2022)\n --no-focus 미러 작업 공간을 선택하거나 창을 활성화하지 않음\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519\n cmux ssh-tmux dev@my-host --transport et --transport-port 8080" } }, "nb": { @@ -87393,14 +87393,14 @@ }, "zh-Hans": { "stringUnit": { - "state": "needs_review", - "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\n通过 SSH tmux 控制模式，将远程主机的 tmux 会话镜像到当前 cmux 窗口中。每个会话成为一个工作区，每个窗口成为一个标签页，多窗格窗口则成为原生分栏。需要启用“Remote tmux”测试功能。\n\n如果需要交互式身份验证，cmux 会在此终端中运行 ssh，然后通过共享连接重试。系统会采用 ~/.ssh/config 中的别名和设置。\n\nFlags:\n --port SSH 端口\n --identity SSH 身份文件路径\n --no-focus 不选择镜像工作区，也不激活其窗口\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" + "state": "translated", + "value": "Usage: cmux ssh-tmux [--port ] [--identity ]\n [--transport ssh|et] [--transport-port ] [--no-focus]\n\n通过 SSH tmux 控制模式（tmux -CC）将远程主机的 tmux 会话镜像到当前窗口的\n侧边栏。每个会话成为一个工作区，每个窗口成为一个标签页，\n每个多窗格窗口成为原生分屏。需要启用\n“Remote tmux”测试版设置。\n\n如果主机需要交互式认证（密码、主机密钥确认、MFA 或触碰安全密钥），\ncmux 会直接在此终端中运行 ssh 以便你完成认证，\n然后通过共享的 SSH 连接镜像会话。无需交互即可认证的主机\n（ssh-agent 或 ~/.ssh/config 中的密钥）\n会直接镜像，不会提示。会遵循 ~/.ssh/config 中的别名及其 IdentityFile/ProxyJump/Port 设置。\n\nFlags:\n --port SSH 端口\n --identity SSH 身份文件路径\n --transport ssh（默认）或 et。et 通过会自动重连的\n EternalTerminal 传输控制流，\n 因此网络变化时镜像会保持而不是重新启动。\n --transport-port 传输连接的端口：ssh 为 sshd 的端口，et 为 etserver 的端口\n （默认 ssh 为 22，et 为 2022）\n --no-focus 不选择镜像工作区，也不激活其窗口\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519\n cmux ssh-tmux dev@my-host --transport et --transport-port 8080" } }, "zh-Hant": { "stringUnit": { - "state": "needs_review", - "value": "Usage: cmux ssh-tmux [--port ] [--identity ] [--no-focus]\n\n透過 SSH tmux 控制模式，將遠端主機的 tmux 工作階段鏡像到目前的 cmux 視窗。每個工作階段會成為工作區，每個視窗會成為標籤頁，多窗格視窗則會成為原生分割。需要啟用「Remote tmux」測試功能。\n\n若需要互動式驗證，cmux 會在此終端機中執行 ssh，然後透過共用連線重試。系統會採用 ~/.ssh/config 中的別名與設定。\n\nFlags:\n --port SSH 連接埠\n --identity SSH 身分檔案路徑\n --no-focus 不選取鏡像工作區，也不啟用其視窗\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519" + "state": "translated", + "value": "Usage: cmux ssh-tmux [--port ] [--identity ]\n [--transport ssh|et] [--transport-port ] [--no-focus]\n\n透過 SSH tmux 控制模式（tmux -CC）將遠端主機的 tmux 工作階段鏡像到目前視窗的\n側邊欄。每個工作階段會成為一個工作區，每個視窗成為一個分頁，\n每個多窗格視窗成為原生分割。需要啟用\n「Remote tmux」測試版設定。\n\n如果主機需要互動式驗證（密碼、主機金鑰確認、MFA 或觸碰安全金鑰），\ncmux 會直接在此終端機中執行 ssh 讓你完成驗證，\n然後透過共用的 SSH 連線鏡像工作階段。不需互動即可驗證的主機\n（ssh-agent 或 ~/.ssh/config 中的金鑰）\n會直接鏡像，不會提示。會遵循 ~/.ssh/config 中的別名及其 IdentityFile/ProxyJump/Port 設定。\n\nFlags:\n --port SSH 連接埠\n --identity SSH 身分檔案路徑\n --transport ssh（預設）或 et。et 透過會自動重新連線的\n EternalTerminal 傳送控制串流，\n 因此網路變更時鏡像會維持，而不是重新啟動。\n --transport-port 傳輸連線的連接埠：ssh 為 sshd 的連接埠，et 為 etserver 的連接埠\n （預設 ssh 為 22，et 為 2022）\n --no-focus 不選取鏡像工作區，也不啟用其視窗\n\nExample:\n cmux ssh-tmux dev@my-host\n cmux ssh-tmux dev@my-host --port 2222 --identity ~/.ssh/id_ed25519\n cmux ssh-tmux dev@my-host --transport et --transport-port 8080" } } }