Unable to connect since recently

[zoqaeski]

I've installed mosh on two Arch Linux machines, both with fully up-to-date software. As of the last few weeks (?), I've been unable to use mosh to remotely access either one from the other, even though plain SSH works just fine. Even if both the laptop and desktop are sitting next to each other on the same desk, and connected via an ethernet switch, any time I run mosh I get the blue bar with "No response received from server" message, before mosh-client exits.

It used to work—the relevant ports in iptables are open on both machines (UDP 60000:61000 and TCP 22)—and disabling iptables has no effect. I'm not sure what else to check to see if I've misconfigured something somewhere, or whether it's a mosh/SSH/iptables/Linux bug.

If anyone can suggest things for me to test, I'd be happy to try it out, as this is confusing me and quite annoying—the main reason I was using mosh was because it would reestablish the connection after the laptop had been suspended, which I do every day to take it to class.

[cgull]

People are reporting what seems like a similar issue in issue #727. Since Arch is a rolling release and will pick up changes quickly, I'm wondering if they're the same issue. You could try the debug steps there to see if they result in useful info. Another thing you can try is looking at mosh-server execution with last:

• Start an ssh connection to the server so you can observe what's going on with mosh
• Start a mosh session.
• Switch to the ssh session and type last $USER | head.
• Wait 60 seconds (this is how long mosh-server takes to time out and exit if it doesn't hear from a client), and type last $USER | head again.

On each invocation of last you should see something like one of these lines as the first line of output (what you see will differ slightly, these examples are on FreeBSD):

jhood      pts/10   ::1 via mosh [11725]   Mon Mar 14 16:23   still logged in
jhood      pts/10   ::1 via mosh [11725]   Mon Mar 14 16:23 - 16:24  (00:00)
jhood      pts/10   mosh [11725]           Mon Mar 14 16:32   still logged in
jhood      pts/10   mosh [11725]           Mon Mar 14 16:32 - 16:33  (00:01)

...or if the problem is similar to that in #727, you might not get anything logged by mosh at all. If you can copy/paste the output you get from each, that'll help us figure out what the problem is.

[cgull]

Oops, that pull request is for #727, not #730.

[cgull]

I can't reproduce this problem on a fresh Arch install on an EC2 instance, either with the Arch package or a build of mosh from source.

You might also try tcpdump on both machines, to see whether or not packets are being received from the other.

[cgull]

No response from OP.

[zoqaeski]

I'm so sorry, I forgot all about this bug because university studies and life caught up with me. I followed up on your debugging suggestions, and it seems that mosh isn't correctly starting mosh-server on the server for some reason. Opening an SSH session, running mosh-server then copying the key to a new terminal and running mosh-client results in a successful connection.

Here's the output from last $USER | head:

zoqaeski   pts/11       192.168.1.108 vi Thu May 12 17:25   still logged in
zoqaeski   pts/11       mosh [30809]     Thu May 12 17:24 - 17:25  (00:00)
zoqaeski   pts/10       mosh [30756]     Thu May 12 17:23 - 17:24  (00:01)
zoqaeski   pts/10       mosh [30673]     Thu May 12 17:20 - 17:21  (00:01)
zoqaeski   pts/12       192.168.1.108    Thu May 12 17:16 - 17:16  (00:00)

The bottom entry is the SSH session, the three mosh ones are attempts to start mosh from the client, then the top one is a successful mosh session after manually starting mosh-server on the server and copying the key to the client. During the failed attempts to start mosh from the client, I thought I'd try watching to see if a mosh process even starts with watch -n 0.5 ps aux | grep mosh, and the only process it listed was the grep process to search ps aux.

EDIT I found out the problem. In /etc/systemd/logind.conf I had "KillUserProcesses=yes" for some reason; the default is no. This was causing systemd to kill the mosh-server process after SSH exited. Not a bug, a feature.

[cgull]

Thanks for letting us know! This appears to be similar to #399, then-- though the exact bit of systemd machinery and configuration appear to be different.

Now that we've had a few different instances of this issue, it's probably worth researching what criteria systemd is finding/killing processes by (process group? session? controlling tty? control group?) and seeing if we can have mosh-server escape the issue.

[bbarenblat]

The officially supported way to have mosh-server escape is to run it with systemd-run. Instead of simply invoking mosh-server new, Mosh should run

systemd-run --scope --user mosh-server new

per the examples in systemd-run(1).

[keithw]

How do screen and tmux deal with this? Do people have to type all that every time they want to start a new screen/tmux session?

[keithw]

(I don't think we're going to be able to change the default command-line, since there's no way for the mosh client to know if the server has systemd or not. But we might be able to change mosh-server to detect this situation and run itself under systemd-run. We should probably just do whatever screen/tmux do on these systems.)

[andersk]

@bbarenblat I can’t get that to work. First I tried it on RHEL 7, which has systemd --user instances disabled. Then I tried it on Ubuntu 16.10 and found that processes inside systemd-run get killed on the user’s last logout even when they otherwise wouldn’t.

[andersk]

@bbarenblat Apparently you’re supposed to run loginctl enable-linger to ask systemd to keep your systemd-run sessions around. But I can’t get that to work either.

The behavior I got when testing this is most charitably described as janky. I saw random 30-second delays on ssh login. I saw systemd get stuck in states where it would refuse to launch systemd --user. I saw systemd suddenly decide to kill all my processes while I was logged in over ssh and not touching anything.

If you can figure out a way to get this to work, I’m interested, but as far as I can tell this functionality isn’t ready for production.

[intelfx]

@andersk

If you can figure out a way to get this to work, I’m interested, but as far as I can tell this functionality isn’t ready for production.

The actual proper way to handle this problem inside mosh would be to create a PAM session for the mosh server. This is the correct thing to do in any case because it is possible for the user to have elaborate PAM configuration that performs arbitrary actions (e. g. pam_mount which mounts user's home directory upon first login and unmounts upon last logout, and this is not even a systemd thing), and AFAIK this too will break with mosh. systemd-logind cleaning up user processes is just a singular example of such configuration.

Now I haven't tried this myself, it's possible there are caveats (e. g. I don't know if it is allowed to create nested PAM sessions) — I mainly want to know if mosh would be at all interested in this kind of a solution (with a corresponding optional dependency on libpam).

[andersk]

@intelfx PAM sessions for a user need to be created by a process running as root, which is not how Mosh works.

[intelfx]

@andersk Oh, right. Indeed. I assume a setuid helper is out of the question?

[achernya]

Closing in favor of #529