Running My Own Email Server (2021-07-29)

Many people will tell you that running your own email server on the internet is crazy, and that the likely result is the email you send will end up in the recipient's spam folder if it is delivered at all. They aren't wrong, running your own email server on the internet is up there with rolling your own crypto in the list of technical things you should never do. While I'm all about defying established convention, there's a twist to this this story that makes it less crazy than it sounds. Let's go on a journey for the reasons behind the change, alternatives I considered, why I chose to run my own email server, and how I configured it.

Background

Prior to running my own email server, I had been using a service called Tuffmail for my email. There were two reasons for this. First, they were one of the cheapest email service providers, costing $24/year. Second, they allowed an unlimited number of email aliases for accounts. I fancy myself a frugal individual, so the low cost appealed to me. I also make heavy use of email aliases. Almost every online service I interact with gets their own email address in my domain. This allows me to easily see which online services have sold the email address to another party, or had their service hacked and had the email address leaked. Setting up the aliases did require logging into Tuffmail's website and clicking around, but generally took only a couple minutes per alias. I currently have about 200 email aliases, though I would guess at least a third I could safely delete.

In terms of interacting with Tuffmail, I used secure POP3 to download received emails, and secure SMTP on the submission port (587) to send emails. Occassionally I used Tuffmail's web interface if I was worried that a message I expected to receive did not arrive, as that allowed me to check the spam folder (POP3 doesn't support that, it only supports downloading messages from the inbox). It was very rare for Tuffmail to have false positives for spam, as I tended to use conservative spam settings.

My Hand is Forced

I had been using Tuffmail for over 10 years, and was quite happy with their service. Unfortunately, earlier this month, Tuffmail sent an email to all of their customers stating they they were ceasing operation on January 1, 2022. So I knew I had to migrate off Tuffmail, and had to decide where to move instead. I'm not one to procrastinate, so I started looking for alternatives quickly.

Deciding Among Alternatives

Tuffmail helpfully included a few alternatives in their email announcing the ceasing of operations, including Fastmail, Proton Mail, Runbox, Greatmail, MX Route, Zoho, Google, and Microsoft. So I looked into each of these options:

• Fastmail: Their $60/year plan supports custom domains, and they support 600 email aliases per address. Fastmail was definitely a possibility.
• Proton Mail: Doesn't support POP3, and requires a separate custom bridge program to support IMAP/SMTP. So Proton Mail was out.
• Runbox: Only supports 100 email aliases per account, with each additional one being about $1/alias. Price was even lower than Tuffmail ($20/year instead of $24/year), but I couldn't deal with the 100 email aliases limit, so Runbox was out.
• Greatmail: Seems to be designed for businesses, smallest plan is $100/year for 8 mailboxes. They support aliases, but I couldn't find documentation on how many. I though $100/year was too much, so Greatmail was out.
• MX Route: I couldn't find documentation on whether they supported aliases, so MX Route was out.
• Zoho: Also designed for businesses, seeminly an equivalent to Microsoft 365. I wanted something focused on email, so Zoho was out.
• Google: Only allows 30 email aliases per user, so Google was out.
• Microsoft: Microsoft 365 supports up to 400 aliases per user, and costs $70/year. However, based on my previous experience with Microsoft 365, it was way too complex for simple email needs. Additionally, from my research Microsoft 365 is known to completely drop emails after accepting delivery, so Microsoft was out.

So of these options, Fastmail was the only service that looked like it would meet my needs. If Runbox allowed for hundreds of email aliases, I would definitely have considered it. As I mentioned, I'm frugal, and $60/year just to send and receive email was more than I wanted to spend. At this point, I considered running my own email server in the cloud.

As I mentioned ealier, running your own email server is kind of crazy, but the actual reason it is crazy is that your email deliverability is probably going to suck. Most people use large email providers such as Gmail, and most large email providers treat email from non-large email providers as more likely to be spam. This is unfortunate but not unexpected, as statistically speaking, such email is more likely to be spam.

This led me to a interesting observation. Running your own email server to receive email is probably fine. You only need to use a large email service to send email. There is no requirement that your service for sending email be the same as the service for receiving email. I could easily setup a virtual machine (VM) in the cloud to receive email, and then find a separate service for sending email.

Bifurcated Email

So my next step was to find a service I could use for sending email. Thankfully, I found one quickly. Sendgrid is a well known service for sending email, and it turns out, they have a free plan that allows for sending 100 emails per day. I read well over 100 emails a day, but I probably only send about 100 emails per month, so Sendgrid's free plan sounded like it would work for me.

So the next step was deciding on a cloud service for hosting a VM to receive email. First, I knew I would be using OpenBSD for this VM. OpenBSD well known for its strong security-focus. It also ships with a mail server (smtpd) with a good security record for my configuration (delivery to maildir). It's what I'm most familiar with, since I use it as an email server in my home network. For full disclosure, I'm also an OpenBSD developer, though I mostly stick with maintaining ports related to Ruby.

So I needed to make sure the cloud provider I selected supported OpenBSD. In terms of performance, my needs were very minimal. A OpenBSD VM just to receive email takes almost no resources. You could probably run it on 64MB of RAM if you turned off kernel and library relinking, and it is hard to find a cloud provider that will let you create a VM with less than 512MB of RAM.

I first considered Amazon Web Services (AWS). AWS supports very small VMs with 512MB of RAM. It looks like the best deal for me would have been a t3a.nano VM, which if you reserve for 3 years, ends up being about $17/year. You need to pay separately for storage, but an OpenBSD installation is very light on storage. It's definitely possible to run with 1GB of storage, and if you want to be safe, maybe 2GB. With 2GB, that would bring the total cost to about $19/year. AWS also gives you the equivalent of a full year of free service for new accounts, so it wouldn't cost anything right away.

I was definitely considering AWS, but the main reason I decided against it is they do not officially support OpenBSD. There are tools you can use to build your own OpenBSD AMI file to use with AWS, but I didn't feel like running something unsupported, at least if I could find a reasonable alternative.

I searched for openbsd cloud and the first thing that popped up was Vultr. They officially support OpenBSD in the cloud and make it very easy to use. The price for Vultr's smallest VM was more expensive than the price for the 3-year reserved t3a.nano instance at Amazon. When I first looked at their service, the smallest VM that I saw they supported was $60/year for 1GB of RAM and 25GB of disk. However, I found that they also offered $50 in credit for new signups, which was almost a full year for the smallest server. I decided to go with Vultr. It was the same cost as Fastmail, but I knew that running my own server would make things like alias creation easier, since I could make it scriptable. Plus, if in the future I could benefit from a VM in the cloud, I would already have one available.

Mail Server Setup

Setting up an OpenBSD VM on Vultr was easy. I was lazy and didn't choose to upload my own ISO, I just picked OpenBSD from their available list of operating systems, then picked the most current version (6.9). Installation took a couple minutes, and I was given the root password and the IP address. From there I sshed into the VM as root and started setting things up.

I first changed the root password using passwd. Next, I setup a user named jeremy for myself using adduser, since I didn't want to be running as root. Then I gave that user access to operate as root using doas (similar to sudo, but simpler). I edited the /etc/doas.conf file:

permit persist keepenv :wheel
permit nopass keepenv root

I also added my public key to /home/jeremy/.ssh/authorized_keys. Then I updated the /etc/ssh/sshd_config file to disallow password authentication and the ability for root to login (leaving other lines in the file alone):

PermitRootLogin no
PasswordAuthentication no

Then I started working on configuring the mail server. To ensure the receiving email server supported STARTTLS for encrypted receipt of email, I followed the instructions in starttls(8):

# openssl genrsa -out /etc/ssl/private/vm.jeremyevans.net.key 4096
# openssl req -x509 -new -key /etc/ssl/private/vm.jeremyevans.net.key \
             -out /etc/ssl/vm.jeremyevans.net.crt -days 10395

Yes, I actually did setup a self-signed certificate that expires in 30 years. Unlike browsers, servers that send email don't generally care if the SSL certificate for the receiving server is self-signed with a long expiration date, since even that is better than sending the email in plain text, which is what the sending server would do if it didn't accept the SSL certificate. If this changes I can always look into getting a certificate signed by a trusted certificate authority.

I edited /etc/mail/aliases so that mail for root would go to jeremy instead (leaving other lines in this file alone):

root: jeremy

I already had a list of all supported email aliases in a text file on my workstation, with one email per line, similar to:

[email protected]

It was easy to convert this to file that smtpd could use, then upload that to the VM:

$ sed 's/$/  jeremy/' < emails-file > aliases-file
$ scp aliases-file vm:emails

Then I edited the /etc/smtpd.conf file on the VM:

pki vm.jeremyevans.net cert "/etc/ssl/vm.jeremyevans.net.crt"
pki vm.jeremyevans.net key "/etc/ssl/private/vm.jeremyevans.net.key"

table local_aliases file:/etc/mail/aliases
table allowed_addresses file:/home/jeremy/emails

listen on lo0 tls pki vm.jeremyevans.net
listen on vio0 tls pki vm.jeremyevans.net

action from_remote_delivery maildir "/home/jeremy/Maildir/Inbox" user jeremy virtual <allowed_addresses>
action from_local_delivery maildir "/home/jeremy/Maildir/Inbox" alias <local_aliases>

match from any for domain "jeremyevans.net" action from_remote_delivery
match from local action from_local_delivery

This configuration listens on both the loopback address (lo0) and the network interface (vio0), and supports STARTTLS on both (though it isn't needed on the loopback). For emails received on loopback, those use the local aliases file (/etc/mail/aliases). For emails received on the network interface, it uses the list of aliases I uploaded, with all of the aliases pointing to jeremy. In both cases, all email is delivered to a single maildir, with a separate file per email. smtpd will create the maildir for me, with the correct ownership and permissions.

For security, I wanted to make sure that only the SSH and SMTP ports were open. I also wanted to allow ICMP so I could ping the server. So I edited /etc/pf.conf:

if = "vio0"
set skip on lo
block return    # block by default
pass in on $if inet proto icmp
pass in on $if inet proto tcp to port {22, 25}
pass out on $if

If I cared more I would probably have setup egress filtering, but I didn't care enough to do that.

Most of the daemons that OpenBSD runs by default are helpful, but in this case, I didn't need to support sound or IPv6, so I disabled the related daemons using rcctl:

# rcctl disable slaacd sndiod

At this point, I remembered it's a good idea to apply the latest security patches using syspatch, so I did that:

# syspatch
# reboot

I then tested with telnet and made sure I could receive an email, and it went through fine, with the email file showing up in /home/jeremy/Maildir/Inbox/new in the VM. Now I needed to get that email file in my local machine, which uses a similar maildir setup. I use mutt for reading the email directly from the maildir. With Tuffmail, I was using secure POP3 to download the email, but I didn't have any experience setting up a POP3 server. However, I determined that wasn't necessary to do that. Since I control both the server and the client, I can just use rsync the files.

Since openrsync would require additional setup, I decided to install rsync on the VM:

# pkg_add rsync

A basic transfer using rsync was fine:

/usr/local/bin/rsync -rt --remove-source-files vm:Maildir/Inbox/new/ ~/Maildir/Inbox/new/

However, this needed to be automated. I had a cron job that ran every 5 minutes to download email from Tuffmail via POP3, and I needed something similar for downloading email from the VM via rsync. However, I don't want the cron job to use my ssh-agent, and therefore decided to setup a separate passwordless SSH key for this using ssh-keygen, and add it to the /home/jeremy/.ssh/authorized_keys file on the VM:

ssh-keygen -t ed25519 -f ~/.ssh/vm-emails_ed25519
cat ~/.ssh/vm-emails_ed25519.pub | ssh vm "sh -c cat >> .ssh/authorized_keys"

Then I updated my ~/.ssh/config file to use this dedicated passwordless key:

Host vm-emails
Hostname vm.jeremyevans.net
IdentitiesOnly yes
IdentityFile ~/.ssh/vm-emails_ed25519

The script to download emails from the VM only changed slightly for the new Host entry:

/usr/local/bin/rsync -rt --remove-source-files vm-emails:Maildir/Inbox/new/ ~/Maildir/Inbox/new/

Then I ran crontab -e to add the cron job:

0,5,10,15,20,25,30,35,40,45,50,55       *       *       *       *       /home/jeremy/bin/download-received-emails

This works well, but it would allow the use of the dedicated passwordless SSH key for general server login, which is a bad idea from a security perspective. So I wanted to lock that down. I found a good tutorial on how restrict SSH key usage for rsync, and then added this to the start of the appropriate entry in /home/jeremy/.ssh/authorized_keys in the VM:

command="rsync --server --sender -tre.iLfxCIvu --remove-source-files . Maildir/Inbox/new/" 

So with all that setup, every 5 minutes my workstation downloads new emails from the VM using rsync over SSH, which I then can read with mutt.

While this was going on, I was still receiving email via Tuffmail, since I hadn't changed my DNS MX record. After a few more tests, I decided to switch my MX record to point to the VM, and things basically continued to work. I immediately tested manually from Gmail and saw the email go through the VM. Within an hour (the TTL for the MX record), all of the incoming email had switched from Tuffmail to the VM.

It's always important to think about backup, so I setup a backup configuration similar to what I use on my home machines. I created an /etc/backup_list file, with paths of the files and folders I want to backup. I ony needed to add things here that I modified after the VM installed, so the file is small:

etc/adduser.conf
etc/backup_list
etc/doas.conf
etc/group
etc/localtime
etc/mail/aliases
etc/mail/smtpd.conf
etc/master.passwd
etc/passwd
etc/pwd.db
etc/pf.conf
etc/pkg_info
etc/rc.conf.local
etc/ssh
etc/ssl/private/vm.jeremyevans.net.key
etc/ssl/vm.jeremyevans.net.crt
home/jeremy/.ssh
home/jeremy/emails
root/.ssh
var/cron/tabs/root

I have a make_backup_tarball script that I use to create a backup file:

BACKUP_LIST=/etc/backup_list
TAR_FILEPATH=/home/jeremy/vm.tar.gz

pkg_info > /etc/pkg_info
tar zcpf $TAR_FILEPATH -C / -I $BACKUP_LIST
chmod 440 $TAR_FILEPATH
chown jeremy:jeremy $TAR_FILEPATH

Sending Email Setup

Setting up Sendgrid was easy. I signed up for a free account, then walked through the process of verifying my domain, which involved adding 3 DNS CNAME records. After that, I created an Sendgrid API key with only Mail Send permissions.

To switch from using Tuffmail to using Sendgrid to send emails, in /etc/mail/smtpd.conf in my local network, I switched:

table secrets file:/etc/mail/secrets
action relay_mail relay host smtp+tls://[email protected]:587 auth 

table secrets file:/etc/mail/secrets
action relay_mail relay host smtp+tls://[email protected]:587 auth 

And updated the /etc/mail/secrets file for the change:

sendgrid apikey:***API_KEY_HERE***

Then I tested sending an email to my Gmail account, and checked that it was correctly send through Sendgrid.

Cost Reduction

While writing this post, I found that Vultr actually offers a cheaper VM. Instead of $60/year for 1GB of RAM and 25GB of storage, you can spend $42/year for 512MB of RAM and 10GB of storage. There's also a $30/year VM for 512MB of RAM and 10GB of storage, if you are willing to switch to IPv6 only. However, I actually want to receive mail, so I cannot use the IPv6-only option. The reason I didn't see this originally is this VM type wasn't available in the Silicon Valley datacenter I picked. I would have to switch from Silicon Valley to New Jersey to get the cheaper VM.

Again, I'm a frugal guy, so saving $18/year seems like a good idea to me. Vultr doesn't let you downgrade VMs, so I spun up a new VM with the $42/year plan, copied the the backup I had already created, than ran the following commands to restore the configuration onto the cheaper VM:

# tar zxpf vm.tar.gz -C /
# pwd_mkdb /etc/master.passwd
# pkg_add rsync
# syspatch
# reboot

The latency when working on the $42/year VM hosted in New Jersey was noticeably worse than the $60/year VM hosted in Silicon Valley. However, if you are also frugal, I'm sure you'll agree that's a small price to pay to save $18/year.

After checking that the $42/year VM worked, I switched the DNS A record to point to it instead of the $60/year VM. After the DNS A record TTL expired, I checked and made sure there was no received email on the $60/year VM, then I turned off and deleted it. I still have the $50 in Vultr credit, which will now last me about 14 months instead of 10. After the free period expires, this will still be more expensive than the $24/year I was paying previously, but it allows me to write a small script to add an email alias, instead of forcing me to login and mess with a web application. Hopefully prices for VMs will continue to fall, though I expect we will soon be at a point where the cost of the IPv4 address is the majority of the cost of a small VM.

Conclusion

I think this bifurcated setup with a cloud VM receiving email and using Sendgrid's free plan for sending email is the cheapest and simplest option that meets my uncommon needs. It is still more expensive than what I was paying Tuffmail ($42/year instead of $24/year), but I suppose I'll manage. I am not currently doing any spam filtering on received email, which results in more spam getting through. That doesn't bother me much, since I can almost always tell what is spam before I open it in mutt, and I just delete spam before opening it. On the plus side, I no longer have to be worried about email being sent to me not arriving because it was marked as spam. Sendgrid is also a larger email provider than Tuffmail, so I'm guessing my email delivery is better, though that doesn't matter much as I don't send much email.