The excellent ISP that I use in Switzerland, Fiber7, has launched a new product in Switzerland with 10 gigabit and 25 gigabit throughput. Unfortunately my current apu2 from PCengines is not powerful enough to route this connection, so I set out to buy a new router. Since I’m really into using open source solutions due to their flexibility, power and low cost — of course I did it with NixOS.

Requirements

I needed something that:

• Routed 10Gbps
• Wasn’t noisy
• Had 2 SFP+ ports
• Wasn’t giant
• Could be operated headless

I went back and forth over a few options and had a short list in the end. What was interesting was while there are some motherboards with support for 2 SFP+, its also not a big deal to use a x16 PCIe slot for a 2 port card. There were also ARM based options, but I limited myself to x86 since I’ve had issues in the past with non-mainstream bootloader situations and them not getting nearly as much attention as x86. Perhaps next time!

The short list was

• Supermicro mini server (various models)
• Mini-ITX sized AMD motherboard from ASrock rack (X570D4I-2T)
• Intel NUC 9 Pro

I ended up choosing the NUC9 pro because:

• It was a much smaller volume than a Mini-ITX sized solution would have been that could also accept a big enough fan to keep the cooling situation quiet.
• It wasn’t made as a rackmount computer first. The Supermicro mini servers unfortunately are made for 1U form factor first, and have really fast noisy fans. A workstation seemed more in line with my type of usage and expectations.

So I put in an order for the NUC9V7QNX (about CHF 800) and waited about a month for it to show up. I don’t know if it’s generally due to the chip shortages or other reasons. Also be careful about pricing — the shortages have driven local pricing up a lot and its worth finding pricing history before choosing.

Physical Setup

The barebones setup came with Ikea-style directions for setting it up. I already had a spare nvme, but had to buy memory and a network card. I ended up choosing the X710DA2 because it was available but ~whatever dual port 10 gigabit card would work here.

I thought the fan lid was pretty cool. It easily came off with just a couple screws. One thing to note is this is not a tool-less case, and requires removal of 6 cross head screws to complete the initial install.

After removing the case sides and a cross bar, one sees the compute element.

After a few more screws are removed, then I was able to install the nvme and memory. Note that there is a power cord for a video card here. I didn’t end up using it but I thought it was a nice touch that it had ‘standard’ and a right angle connector depending on where the connectors are on the card.

Tolerance between the memory and the heatsink is extremely tight. Out of spec memory could be a problem here. Intel has a list of tested memory that might be useful to consult to avoid problems here. Lastly, there are 2 places to put nvme cards and they’ll need to be without heatsinks, as the lid has its own heatsink that presses on them.

Next I installed the network card and tucked the power cable away so it wouldn’t impede airflow near the card. Airflow in this case comes in the mesh sides and is pulled through the top of the case.

And finally a quick look at the back of the case.

Software Setup

I use NixOS for nearly all of my computers and also use it here. My configuration is weird because I use nftables. You should likely go read another doc if you’re really interested in more idiomatic ways to set up routing on NixOS. Otherwise, I’ve set up a pretty boring config featuring dhcpcd and dnsmasq. I also have a backup 5G connection I backhaul to my devices with Wifi. In addition I use Tailscale so this 5G can act as an OOB connection. I’ll perhaps share the config in a future post if folks are interested.

I set up Intel ME/AMT so I can manage the machine over the network at home. I know its been a sore point with various security issues but its also exceedingly convenient for a home network. I use amtterm and Mesh Commander to control the machine remotely as needed. Sadly due to weirdness in the hardware on Intel’s part, an HDMI headless adapter to fake a monitor is recommended so the remote KVM support continues to display after boot. amtterm uses SoL and doesn’t suffer from this, but its nice to have both options.

Most importantly I spent some time using trex to check the performance of the machine. I was able to use the advanced stateful support to test both how well the machine could ipv4 NAT ‘normal’ traffic and abnormal small packet UDP traffic. Advanced stateful is important because it can easily work with NAT routing, as it can adapt to the NAT mapping. I discovered that I needed to set iommu.passthrough=1 as a kernel parameter to gain significant performance on a small packet UDP test. (175kpps increased to over 1Mpps!) The SFR traffic nat test included with trex ran at about 8.5Gbps overnight (as a burn-in test) without significant issues. This test featured over 15k active sessions at a time!

I could have spent more time to characterize the latency under load but decided these breaking points were so far past the sort of usage I planned on that it would not be a problem.

I also tested iperf3 single streams out of curiosity directly to the router (I had already put it in service, oops) and reached well over 9Gbps/sec rates.

I’m not a network benchmarking expert and I’ve likely made mistakes here, but I’m sure that this will do more-than-required for a home network on a 10G connection. I was very surprised to see that without significant tuning work a high performance home router is easily achievable.

Heat and Noise

So far I haven’t noticed the noise. I don’t have any measuring equipment but I don’t really notice it in a quiet room when idle but does make a little noise when fully loaded. A phone a short distance away read over 50dBm but its not really a calibrated device. Its quieter than my Macbook Pro on an hour of video calls, and over that whine I won’t notice anything else anyways!

I was worried that the network card would get too hot but indications are that is not a problem. Remember this was designed for workstation video cards. The exhaust from the CPU is much warmer! A more direct shot of the exaust showed upwards of 47C.

Conclusion

The NUC 9 Pro is a bit overkill but I’m really happy to have a small quiet router ready to go for the day that Init7 installs my new connection.

I’ll write an update in the future when the connection arrives with obligatory speedtests and a longer term opinion.