Linux Fu: The Infinite Serial Port

Ok, the title is a bit misleading. Like most things in life, it really isn’t infinite. But I’m going to show you how you can use a very interesting Linux feature to turn one serial port from a microcontroller into a bunch of virtual ports. In theory, you could create over 200 ports, but the reality is you will probably want to stick with fewer.

The feature in question is what’s known as pseudoterminal or sometimes a pty or pts. These special files were made to feed data to programs that expect to accept data from a terminal. The files provide two faces. To the client, it looks like any other terminal device. To the creator, though, it is just another file. What you write to that file goes to the fake terminal and you can read anything that is sent from the program connected to the terminal. You use these all the time, probably, without realizing it since running a shell under X Windows, for example, doesn’t attach to a real terminal, after all.

You could, of course, do the same trick with a composite USB device, assuming you have one. Also assuming you can find a working driver and get it working. However, many microcontrollers have a serial port — even one with a USB converter built-in — but fewer have full-blown USB hardware. Even the ones that do are often at odds with strange drivers on the PC side. Serial ports work and work well even on the simplest microcontrollers.

The Plan

The plan is simple enough. A Linux program listens to a real serial port and watches for special character sequences in the data stream. Those sequences will allow you to switch data so that the data stream will go to a particular terminal. Data coming back from the terminals will go to the real serial port after sending a sequence that identifies its source.

I built a contrived example because there were certain key features I wanted to test. On the microcontroller, one thread reads an analog quantity and another a digital quantity. The system prints both of these out to separate virtual serial ports. If you press a key in either terminal, the associated data will pause. There’s also a debugging terminal and a command terminal that takes user input to alter things like how often to sample data.

The code, by the way, is on GitHub. If you aren’t using an STM32F411 Blackpill with Mbed, you’ll probably have some work to do. However, keep in mind that the complexity in the code is to fit in with the Mbed libraries. The actual protocol is simple and you can implement it anywhere.

The Protocol

Speaking of the protocol, it is both lightweight and robust. Each device is both a transmitter and a receiver and there is very little connection between them. So while one device is sending to a virtual channel, it might be receiving data from a completely different virtual channel. Each virtual channel has an identification number that can range from 0 to 253.

The transmitter sends most data bytes directly over the port. When there is a desire to change virtual ports, the transmitter sends an FF byte followed by the channel number. There are a few issues to consider.

First, the transmitter has to be aware that if it really wants to send an FF, it can’t. So it sends FF FE instead. If you were sending nothing but FF characters, that would double the amount of data sent, but that’s a relatively rare situation.

Second, there is a robustness problem if the transmitter sends an FF and then dies. The next byte could be considered a channel number. However, since a freshly restarted transmitter should send an initial channel selector, the next byte should be an FF (the initial transmit selector). To combat this, the protocol accepts any number of FF bytes as a legitimate prefix. So all of the following will select channel 4:

FF 04
FF FF 04
FF FF FF FF FF FF 04

The only issue is if the serial connection is intermittent. It is possible to get an FF byte, have the cable go out temporarily and then get a later byte that will trigger an unwanted channel change. There isn’t much you can do about that, although this assumes the next byte is a legitimate channel number since most systems won’t use all possible channels. You could add some robustness by timing out the escape prefix or adding, say, a checksum to the switching sequence but that would only reduce problems not eliminate them. If you are dealing in pure ASCII data, limiting to channels above 0x80 would reduce the issue but, again, not totally eliminate it. In practice, if you have a reliable serial connection, there’s no problem.

Third, there is the case where the receiver dies and starts again midstream. This can cause an issue where some data goes to the wrong virtual terminal. There are two features to help with this. First, if the transmitter sends FF FD to the receiver, the receiver should retransmit its current channel selector. You could also set a transmitter to periodically send the channel selector since it is harmless to send the same selector more than one time.

Another way to combat this partially is to use the -s switch on the Linux server to prevent any data from flowing to the virtual terminals until one is explicitly selected. Note that this only matters during the start-up phase of the server. Once a channel is selected it stays selected until another one is selected.

You can probably see that all the complexity here is to make the server fit in with Linux and the microcontroller side fit in with the C library calls. If you wanted to send data from, say, an Arduino, it would be easy to just prefix everything with “\xFF\xCC” where CC is the channel number. Receiving data could be almost that easy. You just have to remember when you’ve seen an FF and handle the three escape codes (that is, FE is a real FF, FD is a request to resend the channel selector, and everything else is a channel change).

However, I wanted to be able to just have virtual serial ports on both sides of the cable, so it was a lot more work. Part of it is that working with Mbed I/O streams is quirky at best. First, let’s look at the Linux server.

The Linux Server

The Linux software opens a terminal port (e.g., /dev/ttyUSB0 or /dev/ttyACM0 etc.) and then produces multiple pseudo terminals that most terminal software can use. For USB devices, the baud rate is probably unimportant. However, for a real serial device, you probably need to match up baudrates (untested). The code currently does not change the baudrate so you’d need to set it externally or modify the code if you need to.

The ttymux program takes a few options. The only one that is critical is the -c option which defines a virtual port. Each port has an ID number from 0-253. You can also ask for a symlink. So, for example, look at this command:

ttymux -c 10 -c 33:virtualportA -c 50:/tmp/portB /dev/ttyACM0

Here we are creating three ports. Port #10 has no name. Port 33 and 50 will be in ./virtualportA and /tmp/portB, respectively.

Other options:

• -d – Autodelete symlinks on exit
• -n – Do not set attributes on serial port
• -s – Do not send data to a virtual port until expressly selected

When the program runs you’ll see a list of channels and their associated psuedoterminals (probably /dev/pts/X where X is some number). If you don’t provide a symlink, that’s how you connect to the virtual port. If you provide a symlink, you can use either. Note that the ID number is not the same as the pts number. So channel 10 in the above example probably won’t be /dev/pts/10. If it is, that’s just a coincidence.

To compile, you need pthreads:

g++ -o ttymux ttymux.cpp -lpthread

Looking at the code, the server isn’t really that complex. There are two threads. One reads the serial port and the other writes to it. Nothing else touches the real serial port. Then each virtual channel has its own pty. There’s no need to buffer characters or anything.

The part that creates the pty is very simple:

To create the symbolic links, you need to know the name of the pty which is what the getptyname call is for. Using that file name to open will give you the terminal side of the pty. I usually use picocom which has no problems. But some programs, like cutecom, for example, know too much about what a serial port is supposed to look like so you can’t open a pty with them.

The Microcontroller

The microcontroller code is a bit more convoluted. The SerialMux class does all the work. Like the Linux side, it has two threads that control access to the real port. Unlike the Linux side, each virtual serial port object has its own set of input and output buffers. The threads fill those buffers and — eventually — _read and _write do the work of getting the data in and out of the Mbed stream base class. It is not terribly efficient because the streams eventually call _putch and _getch to do I/O character by character, but there are reasons for that and if you really wanted to tweak it, you could override the underlying methods.

One thing that surprised me was that returning true for the isatty function totally breaks the I/O system. I didn’t figure out why, but I did note that the factory-standard USBSerial class also returns false for this and has a comment about it in the Mbed source code.

Another thing that was odd is that the stream’s built in mutex acted strangely and I never figured out exactly why. It may be that it protects all streams or something but I eventually reverted to having my own mutex to protect each set of buffers and another mutex to protect the physical serial port.

Once you follow the logic, the SerialMux class is not that hard. A lot of the code in the main.cpp is to work around issues when the USB port is disconnected.

I also copied and modified a class I’ve used before to build command processing systems. Probably overkill for this, but I had it hanging around and it was easy to use. It is interesting that the code doesn’t know anything about the multiplex system. It just takes a normal stream and works fine.

In Use

Once you have the microcontroller program loaded you can run the Linux server with this command line:

ttymux -s -c 1:analogport.virtual -c 2:digitalport.virtual -c 100:debugport.virtual -c 10:cmdport.virtual

I then use picocom on each virtual port, although you could probably use any other terminal program. You will want to set some options for the command terminal for your own benefit:

picocom -c --omap crlf cmdport.virtual

In the analog and digital terminals, press any key but a space to pause the loop. A space will resume. In the command window, you can use the help command to get a list of what you can do. Don’t expect backspace to work, although there are ways around that if you wanted to fix that.

Future Directions

The protocol is simple even if the example is not and there are many ports possible. Even simple microcontrollers could emulate many ports.

On the server side, someone will surely ask for Windows. It seems like you could do something similar with com0com or one of its related projects. Other ideas would be to feed network sockets instead of pseudo terminals or to work out a system to duplicate fake ports. The protocol and code are simple enough to make any of that possible.

It is true that USB composite devices are a better answer to this problem if you have the hardware and drivers to support it. However, that’s a lot more work, a lot more resources, and limits the ability of cheap hardware to participate.

If you are interested in the USB route, there are some examples of using a Pi Zero and some user-space code if you can provide your own Windows drivers. If the network angle piques your interest, you might have a go with ser2net.