The describes the design and evaluation of Mosh in more detail than you may want. The paper was presented at the 2012 USENIX Annual Technical Conference, held June 13–15, 2012, in sunny Boston, Mass.

In addition, the essay gives further information about the design principles behind Mosh, including the "prophylactic retransmission" technique. The essay was published in USENIX ;login: magazine, August 2012.

Remote-shell protocols traditionally work by conveying a byte-stream from the server to the client, to be interpreted by the client's terminal. (This includes TELNET, RLOGIN, and SSH.) Mosh works differently and at a different layer. With Mosh, the server and client both maintain a snapshot of the current screen state. The problem becomes one of state-synchronization: getting the client to the most recent server-side screen as efficiently as possible.

This is accomplished using a new protocol called the State Synchronization Protocol, for which Mosh is the first application. SSP runs over UDP, synchronizing the state of any object from one host to another. Datagrams are encrypted and authenticated using AES-128 in OCB3 mode. While SSP takes care of the networking protocol, it is the implementation of the object being synchronized that defines the ultimate semantics of the protocol.

Roaming with SSP becomes easy: the client sends datagrams to the server with increasing sequence numbers, including a "heartbeat" at least once every three seconds. Every time the server receives an authentic packet from the client with a sequence number higher than any it has previously received, the IP source address of that packet becomes the server's new target for its outgoing packets. By doing roaming “statelessly” in this manner, roaming works in and out of NATs, even ones that may themselves be roaming. Roaming works even when the client is not aware that its Internet-visible IP address has changed. The heartbeats allow Mosh to inform the user when it hasn't heard from the server in a while (unlike SSH, where users may be unaware of a dropped connection until they try to type).

Mosh runs two copies of SSP, one in each direction of the connection. The connection from client to server synchronizes an object that represents the keys typed by the user, and with TCP-like semantics. The connection from server to client synchronizes an object that represent the current screen state, and the goal is always to convey the client to the most recent server-side state, possibly skipping intermediate frames.

Because SSP works at the object layer and can control the rate of synchronization (in other words, the frame rate), it does not need to send every byte it receives from the application. That means Mosh can regulate the frames so as not to fill up network buffers, retaining the responsiveness of the connection and making sure Control-C always works quickly. Protocols that must send every byte can't do this.

Careful terminal emulation

One benefit of working at the terminal layer was the opportunity to build a clean UTF-8 terminal emulator from scratch. Mosh fixes several Unicode bugs in existing terminals and in SSH, and was designed as a fresh start to try to be robust and correct even for pathological inputs.

Tricky unicode

Only Mosh and the OS X Terminal correctly handle a Unicode combining character in the first column.

ISO 2022 locking escapes

Only Mosh will never get stuck in hieroglyphs when a nasty program writes to the terminal. (See Markus Kuhn's discussion of the relationship between ISO 2022 and UTF-8.)

Evil escape sequences

Only Mosh and GNOME Terminal have a defensible rendering when Unicode mixes with an ECMA-48/ANSI escape sequence. The OS X Terminal unwisely tries to normalize its input before the vt500 state machine, causing it to misinterpret and become unusable after receiving the following input!* (This also means the OS X Terminal's interpretation of the incoming octet stream varies depending on how the incoming octets are split across TCP segments, because the normalization only looks ahead to available bytes.)

* We earlier wrote that this misbehaving sequence "crashes" the OS X Terminal.app. This was mistaken—instead, Terminal.app interprets the escape sequence as shutting off keyboard input, and because of an unrelated bug in Terminal.app, it is not possible for the user to restore keyboard input by resetting the terminal from the menu.

Mosh sets IUTF8

In the POSIX framework, the kernel needs to know whether the user is typing in an 8-bit character set or in UTF-8, because in canonical mode (i.e. "cooked" mode), the kernel needs to be able to delete a typed multibyte character sequence from an input buffer. On OS X and Linux, this is done with the "IUTF8" termios flag.) (See diagnostic explaining the need for this flag.)

Mosh sets the IUTF8 flag when possible and stubbornly refuses to start up unless the user has a UTF-8-clean environment. SSH does not set the IUTF8 flag, which can lead to garbage in input buffers.

Instant local echo and line editing

The other major benefit of working at the terminal-emulation layer is that the Mosh client is free to scribble on the local screen without lasting consequence. We use this to implement intelligent local echo. The client runs a predictive model in the background of the server's behavior, hypothesizing that each keystroke will be echoed at the cursor location and that the backspace and left- and right-arrow keys will have their traditional effect. But only when a prediction is confirmed by the server are these effects actually shown to the user. (In addition, by default predictions are only displayed on high-delay connections or during a network “glitch.”) Predictions are done in epochs: when the user does something that might alter the echo behavior — like hit ESC or carriage return or an up- or down-arrow — Mosh goes back into making background predictions until a prediction from the new batch can be confirmed as correct.

Thus, unlike previous attempts at local echo with TELNET and RLOGIN, Mosh's local echo can be used everywhere, even in full-screen programs like emacs and vi.

Real-world benefits

We evaluated Mosh using traces contributed by six users, covering about 40 hours of real-world usage and including 9,986 total keystrokes. These traces included the timing and contents of all writes from the user to the host and vice versa. The users were asked to contribute "typical, real-world sessions." In practice, the traces include use of popular programs such as the bash shell and zsh shells, the alpine and mutt e-mail clients, the emacs and vim text editors, the irssi and barnowl chat clients, the links text-mode Web browser, and several programs unique to each user.

To evaluate typical usage of a "mobile" terminal, we replayed the traces over an otherwise unloaded Sprint commercial EV-DO (3G) cellular Internet connection in Cambridge, Mass. A client-side process played the user portion of the traces, and a server-side process waited for the expected user input and then replied (in time) with the prerecorded server output. We speeded up long periods with no activity. The average round-trip time on the link was about half a second.

We replayed the traces over two different transports, SSH and Mosh, and recorded the user interface response latency to each simulated user keystroke. The Mosh predictive algorithm was frozen prior to collecting the traces and was not adjusted in response to their contents or results.

The results

Mosh reduced the median keystroke response time from 503 ms to nearly instant (because more than 70% of the keystrokes could be immediately displayed), and reduced the mean keystroke response time from 515 ms to 173 ms. Qualitatively, Mosh makes remote servers "feel" more like the local machine!

Q: Who wrote Mosh?

Mosh was written by Keith Winstein, along with Anders Kaseorg, Quentin Smith, Richard Tibbetts, Keegan McAllister, and John Hood.

Q: Why another remote-terminal protocol?

Practical latency on the Internet is on the increase, with the rise of bufferbloat and sophisticated wireless links that optimize for throughput over delay. And roaming is more common than ever, now that laptops and handheld devices have largely displaced desktops. SSH is great, but frustrating to use when you want to change IP addresses or have a long-delay link or a dodgy connection.

Moreover, TELNET had some good things going for it — a local-echo mode and a well-defined network virtual terminal. Even today, SSH doesn't properly support UTF-8 end-to-end on a POSIX system.

Q: Are the mosh principles relevant to other network applications?

We think so. The design principles that Mosh stands for are conservative: warning the user if the state being displayed is out of date, serializing and checkpointing all transactions so that if there are no warnings, the user knows every prior transaction has succeeded, and handling expected events (like roaming from one WiFi network to another) gracefully.

Those don't seem too controversial, but fancy apps like Gmail-in-Chromium or on Android still behave atrociously on dodgy connections or after switching IP addresses. (Have you ever had Gmail leave an e-mail message in "Sending..." for ten hours while merrily retrieving new mail and not indicating any kind of error? Us too.) We think there may be considerable room for improvement in many network user interfaces from the application of these values.

Q: I'm getting "mosh requires a UTF-8 locale." How can I fix this?

To diagnose the problem, run locale on the local terminal, and ssh remotehost locale. To use Mosh, both sides of the connection will need to show a UTF-8 locale, like LC_CTYPE="en_US.UTF-8".

On many systems, SSH will transfer the locale-related environment variables, which are then inherited by mosh-server. If this mechanism fails, Mosh (as of version 1.2) will pass the variables itself. If neither mechanism is successful, you can do something like

mosh remotehost --server="LANG=en_US.UTF-8 mosh-server"

If en_US.UTF-8 does not exist on the remote server, you can replace this with a UTF-8 locale that does exist. You may also need to set LANG locally for the benefit of mosh-client. It is possible that the local and remote machines will need different locale names. See also this GitHub ticket.

Q: What does the message "Nothing received from the server on UDP port 60003" mean?

This means that mosh was able to start mosh-server successfully on the remote machine, but the client is not able to communicate with the server. This generally means that some type of firewall is blocking the UDP packets between the client and the server. If you had to forward TCP port 22 on a NAT for SSH, then you will have to forward UDP ports as well. Mosh will use the first available UDP port, starting at 60001 and stopping at 60999. If you are only going to have a small handful of concurrent sessions on a server, then you can forward a smaller range of ports (e.g., 60000 to 60010).

Tools like netstat, netcat, socat, and tcpdump can be useful for debugging networking and firewall problems.

This problem can also be the result of a bug in glibc 2.22 that affects programs that link with protobuf and utempter and use aggressive compiler hardening flags. (glibc bugtracker entry, as well as Mosh bugtracker entry.) The problem causes mosh-server to segfault immediately on startup. We believe we have worked around this problem in Mosh 1.2.6, but please report a bug if you find otherwise.

Q: Why do you insist on UTF-8 everywhere?

We're really not UTF-8 zealots. But it's a lot easier to correctly implement one terminal emulator than to try to do the right thing in a variety of difficult edge cases. (This is what GNU screen tries to do, and in our experience it leads to some very tricky-to-debug situations.) So mosh just won't start up until the user has everything configured for a UTF-8-clean pathway. It may be annoying, but it also probably reduces frustration down the road. (Unfortunately an 8-bit vt220 and a UTF-8 vt220 are different and incompatible terminal types; the UTF-8 goes in underneath the vt220 state machine.)

Q: How do I use a different SSH port (not 22)?

As of Mosh 1.2, you can pass arguments to ssh like so:

mosh remotehost --ssh="ssh -p 2222"

Or configure a host alias in ~/.ssh/config with a Port directive. Mosh will respect that too.

Q: I'm getting 'mosh-server not found'.

Please make sure that mosh is installed on the client, and mosh (or at least mosh-server) is installed on the server you are trying to connect to. Also, the server is expected to be available on your server's default login PATH, which is not usually true on OS X and BSD servers, or if you install mosh-server in your home directory. In these cases please see the "Server binary outside path" instructions in the Usage section, above.

Q: SSH authenticates using Kerberos tickets, but Mosh asks me for a password.

In some configurations, SSH canonicalizes the hostname before passing it to the Kerberos GSSAPI plugin. This breaks for Mosh, because the initial forward DNS lookup is done by the Mosh wrapper script. To work around this, invoke Mosh as

mosh remotehost --ssh="ssh -o GSSAPITrustDns=no"

This will often fail on a round-robin DNS setup. In that case it is probably best to pick a specific host from the round-robin pool.

Q: Why is my terminal's scrollback buffer incomplete?

Mosh synchronizes only the visible state of the terminal. We are tracking this issue; see this issue and the others which are linked from there. For now, the workaround is to use screen or tmux on the remote side.

Q: How do I get 256 colors?

Make sure you are running mosh in a terminal that advertises itself as 256-color capable. (This generally means TERM will be xterm-256color or screen-256color-bce.)

Q: How do I type C-^, Mosh's default escape character?

On keyboards with the United States layout, this can be typed as Ctrl-Shift-6, or often as Ctrl-6 (this depends on your OS and terminal emulator). On non-US keyboards, it is often hard to find the right key, and sometimes it's not available at all. If your keyboard has a dead key with an accent-circumflex, this is not likely to be the right key. Ctrl-6 sometimes works, though. If you are unable to type this character, you will need to set the MOSH_ESCAPE_KEY variable; see the Mosh man page for details.

Q: How can I make the server automatically clean up dormant sessions?

Please see the entries for MOSH_SERVER_NETWORK_TMOUT and MOSH_SERVER_SIGNAL_TMOUT in the mosh-server(1) man page.

Q: What is Mosh's security track record so far?

Mosh 1.0 was released in March 2012. As of the release of Mosh 1.3.2 in July 2017, as far as the developers are aware:

• In the last four years, no security vulnerabilities of any kind (major or minor) have been reported in Mosh.

• No major security vulnerabilities have ever been reported in Mosh. We define major security vulnerabilities to include privilege escalation, remote code execution, denial-of-service by a third party, etc.

• Two denial-of-service issues were discovered and fixed in releases in 2012. One issue allowed a mosh-server to cause the mosh-client to spend excess CPU (CVE-2012-2385, fixed in Mosh 1.2.1, released May 2012). Another issue allowed the server host to cause the mosh-client to send UDP datagrams to an incorrect address, foiling its attempt to connect (fixed in Mosh 1.2.3, released October 2012).

Q: How does Mosh's security compare with SSH's?

We think that Mosh's conservative design means that its attack surface compares favorably with more-complicated systems like OpenSSL and OpenSSH. Mosh's track record has so far borne this out. Ultimately, however, only time will tell when the first serious security vulnerability is discovered in Mosh—either because it was there all along or because it was added inadvertently in development. OpenSSH and OpenSSL have had more vulnerabilities, but they have also been released longer and are more prevalent.

In one concrete respect, the Mosh protocol is more secure than SSH's: SSH relies on unauthenticated TCP to carry the contents of the secure stream. That means that an attacker can end an SSH connection with a single phony "RST" segment. By contrast, Mosh applies its security at a different layer (authenticating every datagram), so an attacker cannot end a Mosh session unless the attacker can continuously prevent packets from reaching the other side. A transient attacker can cause only a transient user-visible outage; once the attacker goes away, Mosh will resume the session.

However, in typical usage, Mosh relies on SSH to exchange keys at the beginning of a session, so Mosh will inherit the weaknesses of SSH—at least insofar as they affect the brief SSH session that is used to set up a long-running Mosh session.

Q: Is Mosh affected by the 2018 attacks against the OCB2 cipher mode?

Not that we know of—Mosh uses OCB3. The authors of the paper write that the attack is not applicable to OCB3.

Q: Why does mosh use AES-128 for a session-key, not AES-192 or AES-256?

• AES-128 is more than an adequate key length for a session key.
• The OCB FAQ recommends AES-128.
• AES-128 is a bit nicer and is not subject to the related-key attacks that afflict AES-192 and AES-256. (Schneier: "the key schedule for 256-bit version is pretty lousy -- something we pointed out in our 2000 paper -- but doesn't extend to AES with a 128-bit key." See this blog post.)

Q: Does mosh work with Amazon EC2?

Yes, it works great, but please remember to open up UDP ports 60000–61000 on the EC2 firewall.

Q: How do I tell if mosh is working correctly?

After you run mosh user@server, if successful you will be dropped into your login shell on the remote machine.

If you want to check that mosh is being used instead of ssh, try typing Ctrl-^ Ctrl-Z to suspend the session (with mosh 1.2.4 or later on the client). Running fg will then return.

Q: What's the difference between mosh, mosh-client, and mosh-server? Which one do I use?

The mosh command is a wrapper script that is designed to be the primary way that you use mosh. In most cases, you can simply just replace "ssh" with "mosh" in your command line. Behind the scenes, the mosh wrapper script will SSH to the server, start up mosh-server, and then close the SSH connection. Then it will start up the mosh-client executable on the client, passing it the necessary information for it to connect to the newly spawned mosh-server instance.

In normal usage, mosh-client and mosh-server don't need to be run directly.

Q: How do I run the mosh client and server separately?

If the mosh wrapper script isn't working for you, you can try running the mosh-client and mosh-server programs separately to form a connection. This can be a useful debugging technique.

1. Log in to the remote host, and run mosh-server.

It will give output like:

$ mosh-server 

MOSH CONNECT 60004 4NeCCgvZFe2RnPgrcU1PQw

mosh-server (mosh 1.1.3)
Copyright 2012 Keith Winstein <[email protected]>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 30261]

2. On the local host, run:

$ MOSH_KEY=key mosh-client remote-IP remote-PORT

where "key" is the 22-byte string printed by mosh-server (in this example, "4NeCCgvZFe2RnPgrcU1PQw"), "remote-PORT" is the port number given by the server (60004 in this case), and "remote-IP" is the IP address of the server. You can look up the server's IP address with "host remotehost".

3. If all goes well, you should have a working Mosh connection. Information about where the process fails can help us debug why Mosh isn't working for you.

Q: With the mosh-server on FreeBSD or OS X, I sometimes get weird color problems. What's wrong?

This bug is fixed in Mosh 1.2. Thanks to Ed Schouten and Peter Jeremy for tracking this down.

Q: How do I contribute to mosh?

We welcome your contribution! Please join us in #mosh channel on Libera Chat IRC, visit us on GitHub, or email [email protected]. To contribute to our code base, please fork the repository on GitHub and open a pull request there.

Q: Who helped with mosh?

We're very grateful for assistance and support from:

• Hari Balakrishnan, who advised this work and came up with the name.

• Paul Williams, whose reverse-engineered vt500 state diagram is the basis for the Mosh parser.

• The anonymous users who contributed session logs for tuning and measuring Mosh's predictive echo.

• Nickolai Zeldovich for helpful comments on the Mosh research paper.

• Richard Stallman for helpful discussion about the capabilities of the SUPDUP Local Editing Protocol.

• Nelson Elhage

• Christine Spang

• Stefie Tellex

• Joseph Sokol-Margolis

• Waseem Daher

• Bill McCloskey

• Austin Roach

• Greg Hudson

• Karl Ramm

• Alexander Chernyakhovsky

• Peter Iannucci

• Evan Broder

• Neha Narula

• Katrina LaCurts

• Ramesh Chandra

• Peter Jeremy

• Ed Schouten

• Ryan Steinmetz

• Jay Freeman

• Dave Täht

• Larry Doolittle

• Daniel Drown

• Timo Juhani Lindfors

• Timo Sirainen

• Ira Cooper

• Felix Gröbert

• Luke Mewburn

• Anton Lundin

• Philipp Haselwarter

• Timo J. Rinne

• Barosl Lee

• Andrew Chin

• Louis Kruger

• Jérémie Courrèges-Anglas

• Pasi Sjöholm

• Richard Woodbury

• Igor Bukanov

• Geoffrey Thomas

• Steve Dignam

• HIGUCHI Yuta

• Baruch Siach

• [email protected]
  Mosh development and discussion.

  Sign up or view archives at https://mailman.mit.edu/mailman/listinfo/mosh-devel.

• [email protected]
  Mosh user discussion and site best practices.

  Sign up or view archives at https://mailman.mit.edu/mailman/listinfo/mosh-users.

• #mosh channel on Libera IRC
  

  You can connect with a Web client, try an irc:// URL, or manually configure your client for irc.libera.chat.

• At the recommendation of the security community, confidential security-related matters may be sent to: [email protected]

  Messages may optionally be encrypted with Keith Winstein's public key:

  pub   rsa4096 2012-02-05 [SC] [expires: 2022-02-18]
        B1A4 7069 121F 6642 BB3D  7F3E 20B7 283A FE25 4C69