A cautionary tale about locking Linux & FreeBSD user accounts

A cautionary tale about locking Linux & FreeBSD user accounts

Like every other solo developer and sysadmin, I do stuff using ssh. Some stuff is automated using scripts, and others require ssh login. For example, one of my scripts logs into my Linux and FreeBSD server using public ssh keys and does a particular type of work for me. I have a dedicated user account for that purpose called autovivek on Raspberry PI 4 for Ansible and custom script automation. Here is how it works:
{rpi4:~}$ ssh [email protected] /path/to/taks1
In other cases, it sends scripts and then executes them on the remote server named 192.168.2.17. Sounds good, right? So, when I need to make backups and other tasks, I lock down the autovivek user account on the server so that it will not modify data on disks. For example, here is how to lock down a user account:
{linux-server:~}$ sudo usermod -L -e 1 autovivek
## OR ##
{freebsd-server:~}$ sudo pw lock -n autovivek

A cautionary tale about locking Linux and FreeBSD user accounts

However, I soon discovered that a user named autovivek can still log into the server and make changes despite being locked down on both Linux and FreeBSD servers. I foolishly assumed that it would work out of the box. But, boy, I was in for a big surprise.

Upon close inspection on autobox (RPi 4 computer), I found that I enabled ssh multiplexing for ssh speed up a couple of days back in my ~/.ssh/config file:

Host *
	IdentitiesOnly yes
	ControlPath ~/.ssh/controlmasters/%r@%h:%p
	ControlMaster auto
	ControlPersist yes

What is SSH multiplexing?

SSH multiplexing is nothing but carrying multiple SSH sessions over a single TCP connection. In other words, OpenSSH reuses an existing TCP connection for multiple concurrent SSH sessions. A locked or unlocked user account doesn’t matter on the remote machine if a session is already established on the OpenSSH server. It will directly connect using the ssh. Here is how to find out if the session is open or not:
{rpi4:~}$ ssh -O check [email protected]
Here is what I see:

Master running (pid=418156)

Of course, the ss command will confirm open session too:
{rpi4:~}$ ss -o state established '( dport = :ssh or sport = :ssh )' | grep 192.168.2.17
Session opened in the background

tcp      0         0             192.168.2.25:ssh          192.168.2.17:22201    timer:(keepalive,1min13sec,0)                                                  
tcp      0         0             192.168.2.25:46412        192.168.2.17:ssh      timer:(keepalive,67min,0) 

On the server-side, there will be a similar session open in the background too:
{linux-server:~}$ ss -o state established '( dport = :ssh or sport = :ssh )' | grep 192.168.2.17
Outputs:

Netid                            Recv-Q                            Send-Q                                                       Local Address:Port                                                       Peer Address:Port                             Process                            
tcp                              0                                 0                                                            192.168.2.17:ssh                                                        192.168.2.25:41884                             timer:(keepalive,29min,0)    

In other words, ss command confirmed that ssh listens in the background on the server when ControlPersist set to yes on client side in the ~/.ssh/config file. It keeps the connection open between the ssh client and server to speed up the operation. And that is the root cause of my issue. Ah, the mystery solved. From the ssh_config(5) man page:

1. The ControlPersist directive specifies that the master connection should remain open in the background (waiting for future client connections) after the initial client connection has been closed.
2. If set to no, the master connection will not be placed in the background and will close as soon as the initial client connection is closed.
3. One can set it to yes or 0, then the master connection will remain in the background indefinitely until killed or closed via a mechanism such as the “ssh -O exit“.
4. If set to a time in seconds, or a time in any of the formats documented in sshd_config(5), then the backgrounded master connection will automatically terminate after it has remained idle with no client connections for the specified time.

So, here is how I solved my problem.

How to lock Linux and FreeBSD user accounts safely and block ssh access immediately

A locking account is not enough. It would be best to stop all processes owned by the user named ‘autovivek’. In other words, I do this now:
{linux-server:~}$ sudo killall -STOP -u autovivek
In other words, the correct procedure to lock down the account on both Linux and FreeBSD servers is as follows:
{linux-server:~}$ sudo usermod -L -e 1 autovivek # Linux
{freebsd-server:~}$ sudo pw lock -n autovivek # FreeBSD
Then on both servers:
{linux-server:~}$ killall -STOP -u autovivek # STOP all processes owned by the 'autovivek' user
Of course, you can kill it, but I just needed to stop them and then when the backup is done, I unlock the user account and resume the session:
{linux-server:~}$ sudo usermod -e -1 -U autovivek # Linux
{freebsd-server:~}$ sudo pw unlock -n autovivek # FreeBSD
Again, run the killall command on both servers to resume all processes owned by the ‘autovivek’ user:
{linux-server:~}$ killall -CONT -u autovivek # Resume session

SIGSTOP and SIGCONT signal

The killall command sends the -STOP (SIGSTOP) signal to stop all processes owned by a user named ‘autovivek’. Similarly, the -CONT (SIG) signal is used to continue (resume) the processes owned by a user named ‘autovivek’ if it is currently stopped.

Summing up

Indeed it is a cautionary tale about locking Linux and FreeBSD user accounts. No matter how careful you are, there will always be some side effects with optimization. I enabled ssh speed optimization via multiplexing, and I met with some surprises. Check the following manual pages using the man command for more info:
$ man sshd_config
$ man ssh_config
$ man ssh
$ man kill
$ man killall
$ man 7 signal
So this reminds me of the Unix account removal procedure script I developed many moons ago when the sysadmin deleted the user account on HP-UX. First, it will search for users’ all cron/at jobs, processes, emails and files owned by the user. Then archive all files and delete them.